# Should icon search use embeddings or keywords?

> Keywords answer fast and offline but fail on unfamiliar words. Embeddings catch meaning but cost a request. Run words first and vectors second.

URL: https://semanticicons.com/blog/embeddings-or-keywords-for-icon-search
Published: 2026-10-14
Updated: 2026-10-14

The question is usually asked as a choice. It is not one. The two methods fail in different places, and the failures barely overlap.

## What each one is good at

**A lexical ranker** reduces the query to stems and scores every icon on phrase, keyword and category hits, weighted by term rarity. It runs in the page over a static index. No request, no key, no latency, and the same query always gives the same answer.

It fails when the reader's words are not in your set at all. `a filibuster` matches nothing, however good the tags are.

**A vector search** embeds the query and finds the nearest icon by cosine distance. It matches meaning, so it can answer with **stopwatch** for a phrase that never says *time*.

It fails on precision. Everything has a nearest neighbour, so it always returns something, and it is confidently wrong when nothing in the set is close. It also costs a network request, a key, and a model that must never change without re-embedding the whole set.

| | Lexical | Vector |
| --- | --- | --- |
| Latency | None, runs locally | A network request |
| Predictable | Yes | Mostly |
| Unfamiliar words | Fails | Often works |
| Confidently wrong | Rare | Common without a floor |
| Cost per query | Zero | Small but real |

## Run them in order

The design that works is a cascade with a measured threshold.

1. **Score the query lexically.** If the best score is above the floor, answer with it and stop. This is most queries.
2. **Below the floor, embed the query** and search the vectors. If the nearest icon is above a similarity floor, answer with it.
3. **Below both floors, draw something new**, or say you have nothing.

Our floors are `38` for the lexical score and `0.62` cosine similarity for the vector stage. Both were measured, not chosen. The lexical floor came from a labelled set of 64 prose phrases: it is the point below which the top answer stops being right.

The important property is that **stage two only runs when stage one is not confident**. So the common case costs nothing, and the key is only needed for the hard queries.

## Embed the right text

A vector over an icon needs text to embed, and the obvious choice, the icon's name, is the wrong one. Embed everything the icon knows:

```ts
[icon.phrase, icon.category.replace(/-/g, ' '), icon.concept, icon.keywords.join(', ')].join('. ')
```

This matters more than the model choice. Thin tags make thin vectors, so the work described in [how many keywords does an icon need](/blog/how-many-keywords-does-an-icon-need) improves both layers at once.

## Store the model id with the vector

This is the failure that bites people six months later.

Vectors from one model are meaningless to a query embedded by another. If you switch model, or a provider silently updates one, every stored vector becomes noise that still returns confident results.

Write the model id on every row and filter on it:

```sql
select ... from icons
where embedding_model = $model_id
order by embedding <=> $query
```

Now a model change makes the vector stage find *nothing*, and the cascade falls through to the next stage. That is the correct failure: it degrades to the previous behaviour instead of returning nonsense.

## Which model

For retrieval, pick the model with the better retrieval benchmark, not the one that is easiest to sign up for. We got this wrong once, chose the convenient provider, and reversed it. The cost difference across 3000 icons was about two cents. Convenience should never have outranked recall at that price.

Whatever you pick, keep the same model for the documents and for the queries, and re-embed the whole set when you change.

## Questions people ask

### Do I need a vector database?

Not for a few thousand icons. pgvector with an HNSW index handles this comfortably, and you probably already run Postgres. A dedicated vector store earns its place at a much larger scale.

### How do I choose the similarity floor?

Take twenty queries you expect to answer and twenty you expect to miss. Record the top similarity for each. The floor sits in the gap. If there is no gap, the embedding text is too thin.

### Can embeddings replace keywords entirely?

Only if you accept a request on every search, a confident wrong answer whenever the set has no match, and no offline use. For an icon picker in an editor, none of those is acceptable.

### How often do I need to re-embed?

When the model changes, and when an icon's phrase, concept or keywords change. Adding an icon only needs one new vector.