Sentence Embedding:
Sentence Transformer can be used to embed sentences into a vector space. This is useful for tasks such as text classification or semantic similarity where sentences need to be compared.
from sentence_transformers import SentenceTransformer,util
model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = ['This framework generates embeddings for each input sentence',
'Sentences are passed as a list of string.']
embeddings = model.encode(sentences)
for sentence, embedding in zip(sentences, embeddings):
print("Sentence:", sentence)
print("Embedding:", embedding)
print("")
Sentence Similarity:
Sentence Transformers can be used to compute the similarity between two sentences. This is useful for tasks such as information retrieval, where you need to find documents similar to a given query.
from sentence_transformers import SentenceTransformer,util
model = SentenceTransformer('all-MiniLM-L6-v2')
emb1 = model.encode("I am eating Apple")
emb2 = model.encode("I like fruits")
cos_sim = util.cos_sim(emb1, emb2)
print("Cosine-Similarity:", cos_sim)
Semantic Search:
Sentence Transformers can be used to perform semantic searches. This is useful for tasks such as question answering, where you must find documents containing answers to a given question.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('clips/mfaq')
question = "How many models can I host on HuggingFace?"
answer_1 = "All plans come with unlimited private models and datasets."
answer_2 = "AutoNLP is an automatic way to train and deploy state-of-the-art NLP models, seamlessly integrated with the Hugging Face ecosystem."
answer_3 = "Based on how much training data and model variants are created, we send you a compute cost and payment link - as low as $10 per job."
query_embedding = model.encode(question)
corpus_embeddings = model.encode([answer_1, answer_2, answer_3])
print(util.semantic_search(query_embedding, corpus_embeddings))
Clustering:
A sentence Transformer can be used to cluster documents. This is useful for tasks such as topic modeling or document classification, where you need to group documents by topic or category.
from sklearn.cluster import KMeans
import numpy as np
embedder = SentenceTransformer('all-MiniLM-L6-v2')
corpus = ['A man is eating food.',
'A man is eating a piece of bread.',
'Horse is eating grass.',
'A man is eating pasta.',
'A Woman is eating Biryani.',
'The girl is carrying a baby.',
'The baby is carried by the woman',
'A man is riding a horse.',
'A man is riding a white horse on an enclosed ground.',
'A monkey is playing drums.',
'Someone in a gorilla costume is playing a set of drums.',
'A cheetah is running behind its prey.',
'A cheetah chases prey on across a field.',
'The cheetah is chasing a man who is riding the horse.',
'man and women with their baby are watching cheetah in zoo'
]
corpus_embeddings = embedder.encode(corpus)
corpus_embeddings = corpus_embeddings / np.linalg.norm(corpus_embeddings, axis=1, keepdims=True)
clustering_model = KMeans(n_clusters=4)
clustering_model.fit(corpus_embeddings)
cluster_assignment = clustering_model.labels_
print(cluster_assignment)
clustered_sentences = {}
for sentence_id, cluster_id in enumerate(cluster_assignment):
if cluster_id not in clustered_sentences:
clustered_sentences[cluster_id] = []
clustered_sentences[cluster_id].append(corpus[sentence_id])
print(clustered_sentences)
Learn How to use Sentence Transformers to perform Sentence Embedding, Sentence Similarity, Semantic search, and Clustering.
I am a Freelance Data Scientist working on Natural Language Processing (NLP) and building end-to-end NLP applications.
I Share Practical hands-on tutorials on NLP and Bite-sized information and knowledge related to Artificial Intelligence.
https://www.linkedin.com/in/pradipnichite/
