If you build anything that decides "is this content about that thing" using embeddings — auto-tagging, related content, routing to a forum, spam scoring — the obvious design is a threshold. Score the item against each candidate, keep whatever clears a number, take the top few. It is one setting, it is easy to explain to an administrator, and it is wrong.
The failure
An auto-tagging application shipped with exactly that. On the community that installed it, it gave sixty consecutive topics the same three tags. Its threshold was 40%, its limit was three tags, and it hit the limit every single time.
The tag list was the reason:
Invision Application Flarum Extension Premium Flarum Extension Flarum Theme Premium Flarum Theme
Reasonable tags for a site that sells those things. Measuring the stored tag vectors against each other:
0.827 Premium Flarum Extension | Premium Flarum Theme 0.823 Flarum Extension | Premium Flarum Extension 0.815 Flarum Theme | Premium Flarum Theme 0.747 Flarum Extension | Premium Flarum Theme 0.732 Invision Application | Premium Flarum Extension 0.701 Flarum Extension | Flarum Theme 0.684 Invision Application | Premium Flarum Theme 0.682 Flarum Extension | Invision Application 0.669 Flarum Theme | Premium Flarum Extension 0.590 Flarum Theme | Invision Application
The least similar pair of tags on the whole site is 0.590. The vocabulary is one tight cluster, so any topic relevant to one tag is relevant-looking to all five, and no threshold separates them. Below ~0.59 everything is applied; above the best score nothing is. There is no value in between that behaves.
Why raising the threshold does not help
Cosine similarity has no absolute meaning. Its scale depends on the embedding model, on whether you embedded symmetrically or as query/document, on how long each side is, and on how tightly clustered your candidate set happens to be. A number tuned on one community is meaningless on the next.
Specifically, comparing a short label against a whole topic is asymmetric: those scores sit far lower than the 0.85+ you would expect from comparing two topics. So the developer picks a low threshold to make anything match at all — and a low threshold on a clustered vocabulary admits everything.
What to do instead
Keep the threshold, but demote it to a backstop, and add two relative tests. Sort candidates by score, then:
1. Floor. If the best candidate does not clear the threshold, the item is about something your candidate set does not cover. Return nothing. This is the only job an absolute number can do honestly: distinguishing "no match" from "some match".
2. Dominance. Accept the best candidate. Accept a second only if it scores close to the first — a proportion of the best score, not a fixed distance:
if ( $accepted and $row['score'] < $best * $dominance ) // 0.95 works well
{
continue; // sorted descending, so everything after this fails too
}
This is what stops the "and whatever else cleared the bar" behaviour. Note it is scale-free: it does not care what range the model's scores occupy.
3. Distinctness. This is the one that actually fixes a clustered vocabulary, and it is the one people leave out. Compare the candidate against the candidates already accepted, not against the content:
foreach ( $accepted as $already )
{
if ( Vector::cosine( $row['vector'], $already['vector'] ) >= $distinct ) // 0.70
{
continue 2; // a competing label, not an additional one
}
}
"Premium Flarum Extension" and "Flarum Extension" are 0.823 alike. They are two answers to one question, and applying both is always wrong regardless of how the content scored. Because this measures candidate-to-candidate, it is independent of the item, of the model's scale, and of any per-community tuning — which is what makes it the reliable gate.
On the vocabulary above, these three take three tags per topic down to one, and it is the right one. On a vocabulary of genuinely separate subjects — billing, mobile app, server setup — pairwise similarity runs 0.1–0.3, the distinctness gate never fires, and genuinely multi-topic items still get multiple labels.
Two things worth building alongside it
Show the rejects. A wrong label and a right one look identical once applied, so a screen that lists only the winners cannot be used to diagnose anything. List every candidate with the reason it was kept or dropped — "too similar to Flarum Extension (82%)". In the case above the administrator had no way to tell whether the threshold, the model or the vocabulary was at fault, because the preview screen only showed successes.
Surface the overlap. The real cause was not in the code, it was in the tag list, and the person who created those tags had no reason to think of them as a problem. Comparing your candidate set against itself is cheap and needs no API call, so show which pairs overlap and by how much. It explains the behaviour before somebody concludes the app is broken.
And make bulk classification reversible. Anything that labels content in bulk will get it wrong in bulk at least once; if the only remedy is editing items one at a time, that is the real defect.
Testing it without an API key
You can build vectors with any pairwise similarities you like — Gram-Schmidt over the target Gram matrix — which means a real production failure can be reproduced exactly, with no network and no key, from nothing but the similarity numbers you measured:
/* each vector is placed so its dot product with every earlier one is the
number asked for; unit length, so dot product IS cosine */
for ( $j = 0; $j < $i; $j++ )
{
$dot = 0.0;
for ( $d = 0; $d < $j; $d++ )
{
$dot += $v[ $d ] * $out[ $j ][ $d ];
}
$v[ $j ] = ( $target[ $j ] - $dot ) / $out[ $j ][ $j ];
}
$v[ $i ] = sqrt( 1.0 - $used ); // $used = sum of squares placed so far
🚨 Do not use a candidate's own vector as the content vector. That scores 1.00, the dominance gate then rejects everything else on its own, and the distinctness gate never runs — so the test passes without testing the thing it exists to test. The first version of this test did exactly that and reported success. Build the content vector as a weighted mixture of the candidates plus an orthogonal component standing in for everything else in the text, and solve for how much orthogonal signal reproduces the score profile you actually observed. Real content lands in the middle of the cluster; that is the case that breaks.
Verified against
Invision Community 5.0.19, with Voyage voyage-3.5-lite embeddings. The similarity figures are measured from a live community's stored tag vectors.
Related application: Auto Tagging — Auto Tagging scores candidates relative to the best match rather than on a fixed threshold alone, and lists the tags in a vocabulary that overlap too closely to be told apart.
Recommended Comments