Byte64

Tag: doc ids

  • BM25 & Search Index Encoding

    —

    by

    Andrew
    in ,

    Andrew

    Okapi BM25 is a standard ranking formula that has been used in search engines since the 1980s. For each word in a query, it uses the frequency of that word in a document, the length of the document and the number documents that contain the word to decide how significant the word is for the…

  • Search Indexes and Memory

    —

    by

    Andrew
    in ,

    Andrew

    I’ve been working on v0 of the search engine which requires building a search index in the form of “posting lists”. I’ve build a pipeline that reads wikipedia documents, outputs all the words it finds and emits key-value pairs of (word, url). Then we group by word so we can lookup all documents that word…