Tag: scoring
-
Hybrid Search
I’ve already written a post on using a piecewise-linear scaling to bring BM25 and my semantic score (from cosine similarity with our embeddings) into the same numerical space. After performing this scaling, I found some important results weren’t scoring as well as they should. In particular, any search query with a common term (e.g. “the”…
-
Score Normalization
Currently, I have two different scoring functions: BM25 and the semantic scoring function that comes from our sentence embedding. These scores take very different ranges, but need to be combined to make a final score. It’s not simply a matter of assigning different weights to these scores. We need to stretch them out to make…
-
BM25 & Search Index Encoding
Okapi BM25 is a standard ranking formula that has been used in search engines since the 1980s. For each word in a query, it uses the frequency of that word in a document, the length of the document and the number documents that contain the word to decide how significant the word is for the…
