Why icon search returns the wrong icon
Why does icon search return the wrong icon?
Usually because one icon carries a tag that belongs to another icon. A set with generous keywords creates collisions, and a naive scorer lets the icon with the most tags win every time. The fix is to cap how many keywords of one icon can pay for the same word.
We added keywords to 3000 icons and the search got worse.
Not everywhere. On most queries it improved sharply. But a set of queries that used to work started returning nonsense. Searching put the kettle on returned cup of tea, not kettle. Searching for a phrase that was an icon's own name returned a different icon entirely.
This is a predictable failure, and it is worth writing down, because the obvious fix is the wrong one.
The obvious fix is wrong
The obvious fix is to find the offending tags and delete them. Cup of tea should not claim kettle, so remove it.
We counted. 38 icons were being shadowed this way. Deleting the tags would have fixed those 38 and left the mechanism in place, so the 39th would appear the next time somebody added a keyword. Worse, some of those tags were correct: a cup of tea genuinely does relate to a kettle. The tag was not wrong. The scoring was.
The four faults
One icon paid many times for one word
An icon carrying kettle, boil the kettle, kettle on and electric kettle scored four keyword hits for the single query word kettle. The icon actually named kettle scored one phrase hit. Four beat one.
The fix caps the number of keyword hits one icon can collect for a single term:
const MAX_KEYWORD_HITS = 2;An icon can still be found by any of its tags. It just cannot stack them.
A whole phrase did not outrank a keyword
When a query contains another icon's complete phrase, that icon should win. It did not, because an exact keyword match paid nearly as much as an exact phrase match. The fix shades the keyword bonus down, from 100 to 30, when the query names some other icon's whole phrase.
A partial match could win on intensity
An icon that matched one word very strongly could outrank an icon that matched four words moderately. Search is about covering the query, so coverage needs a floor and a curve:
const COVERAGE_FLOOR = 0.5;
const COVERAGE_POWER = 1.6;An icon covering half the query keeps half its score. Below that it falls away fast.
A term counted twice
An icon whose name repeats a word in its phrase counted that word twice. Removing duplicates before scoring is one line, and it removed a whole class of quiet wins.
The result
We measure retrieval against 64 labelled prose phrases. Each names an icon that a reader would expect. The score is the number returned at rank one.
| Change | Correct, out of 64 |
|---|---|
| Before keywords | 50 |
| Tuning the scoring weights alone | 51 |
| After keywords, before the four fixes | 50 |
| After the four fixes | 63 |
The middle row is the one to look at. A full grid search over every scoring weight bought one answer. The structural fixes bought thirteen.
The one we did not fix
the quiet after everyone leaves returns leaf. The stemmer folds leaves onto leaf, and the noun wins. Fixing it needs part of speech tagging, which is a large amount of machinery for one phrase in sixty-four. We wrote it down as a known limit instead. Not every failure is worth a mechanism.
What to take from it
If your icon search is wrong, measure before you edit. Build a list of thirty or forty phrases with the answer you expect, and run it on every change. Without that list you cannot tell whether a change helped, and you will spend your time on the formula when the fault is in the data, or on the data when the fault is in the formula.
Try the same phrases on our set and see the score for each result.
Questions people ask
How many test phrases do I need?
Thirty is enough to catch a regression. Sixty is enough to tune. Write them as a reader would say them, not as keywords, and record the icon you expect before you run the search.
Should I just use embeddings instead?
Embeddings solve a different failure: a query whose words appear nowhere in the set. They do not fix a shadowed phrase, because the shadowing icon is genuinely close in meaning. Use both, with words first.
Does capping keyword hits lose real matches?
No, in our measurement. An icon still matches on any of its tags. The cap only stops one icon from collecting the same word repeatedly, which was never evidence of a better match.
How do I find shadowed icons?
Search for every icon by its own exact phrase and record the ones that do not come back first. That count should be zero. Ours is.