← All posts

TF-IDF vs Embeddings for Support Search

TF-IDF matches words; embeddings can connect related phrasing. A practical comparison with Python examples and a support-agent fallback design.

By Sahil Khan ·

Retrievalscikit-learnEmbeddings

“Where is my refund?” and “The money hasn’t come back yet” may describe the same problem. They share very few useful words. That is a good starting point for understanding the difference between keyword-based search and embedding search.

In my support-agent project, both approaches exist. Embeddings are the preferred path; TF-IDF is the fallback when the embedding index cannot be built. This is a comparison of how they work, not a claim that I have measured a winning model.

TF-IDF: give distinctive words more weight

TF-IDF turns documents into vectors based on their words. A word that is useful for distinguishing one document from others gets more weight than a word that appears everywhere. Search compares the query vector with the stored document vectors.

The TF-IDF path
  1. Past messagesBuild a vocabulary from their words
  2. TF-IDF vectorsWeight the words in each message
  3. New queryTransform using the same vocabulary
  4. Similarity rankingReturn the closest stored messages

Here is a runnable toy example. Install scikit-learn, then run the snippet in Python. These are invented messages for explaining the method, not customer data:

python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

messages = [
    "Track a missing package",
    "Check the status of a refund",
    "Cancel a subscription",
]
vectorizer = TfidfVectorizer(ngram_range=(1, 2))
index = vectorizer.fit_transform(messages)
query = vectorizer.transform(["Where is my refund?"])
scores = cosine_similarity(query, index)[0]
print(messages[int(scores.argmax())])

The result is “Check the status of a refund”. That is a useful baseline with very little machinery. But a query made entirely of unseen words produces no useful vocabulary overlap. A highest-ranked result can still have a score of zero, so always look at the score as well as the position.

The project's index uses unigrams and bigrams, an English stop-word list, and a minimum document frequency. Those choices affect which terms survive. In a tiny example corpus, copying a minimum frequency of two could throw away the very words you care about.

Embeddings: compare learned representations

An embedding model converts text into a dense list of numbers. A model trained for semantic similarity can place related phrases near each other even when the wording differs. “Refund” and “money back” may therefore connect more easily.

The embedding path
  1. Past messagesEncode them with a trained model
  2. Stored embeddingsKeep the vectors alongside their messages
  3. New queryEncode with the compatible query setup
  4. Similarity rankingCompare vectors and return candidates

Here is an illustrative counterpart using sentence-transformers. The first run downloads a model; it is not an offline-only example:

python
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
messages = ["Check refund status", "Track a parcel"]
index = model.encode(messages, convert_to_tensor=True)
query = model.encode("Has my money come back?",
                     convert_to_tensor=True)
scores = util.cos_sim(query, index)[0]
print(messages[int(scores.argmax())])

This demonstrates the API, not a benchmark. A short list with an obvious answer says little about performance on ambiguous support requests. Embeddings can also retrieve something that sounds related while missing an important detail, such as a cancelled order versus a cancelled subscription.

Why keep both?

TF-IDF does not need a downloaded embedding model. Once its Python dependencies and corpus are available, it is straightforward to build locally. Embeddings add model files, startup work, and another dependency to manage.

The project's fallback
  1. Try semantic retrievalReuse or build an embedding index
  2. If index building failsRecord the failure
  3. Use TF-IDFReuse or build the lexical index
  4. Keep the method visibleApply the matching decision threshold

This is failover, not hybrid search. The current implementation does not merge rankings from both methods. Likewise, the TF-IDF snippet above is a simplified example; the repository persists its vectorizer and matrix for reuse.

How I would compare them properly

I would collect representative queries, mark which stored examples are useful, and run both retrievers on the same set. Then I would inspect the top few results, including paraphrases, short messages, and requests with no good match. I would keep those test queries separate from examples used to tune thresholds.

The important question is whether useful evidence reaches the next stage. A higher cosine score from one method does not establish that it is better than the other, and neither score proves the final reply is correct.

For the surrounding workflow, see Building an AI Support Agent.

Code and further reading