Byte64

Try the wikipedia search demo here.

Recent Blog Posts

  • Score Normalization

    —

    by

    Andrew
    in

    Currently, I have two different scoring functions: BM25 and the semantic scoring function that comes from our sentence embedding. These scores take very different ranges, but need to be combined to make a final score. It’s not simply a matter of assigning different weights to these scores. We need to stretch them out to make…

  • BM25 & Search Index Encoding

    —

    by

    Andrew
    in ,

    Okapi BM25 is a standard ranking formula that has been used in search engines since the 1980s. For each word in a query, it uses the frequency of that word in a document, the length of the document and the number documents that contain the word to decide how significant the word is for the…

  • Vector Retrieval

    —

    by

    Andrew
    in , ,

    Now that every document has been assigned a vector encoding its semantics, this opens the door to a new kind of retrieval. Rather than find documents that might be relevant to the query by searching through the search index for keywords, we can instead take the query’s vector and find nearby documents in the embedding.…

  • Vector Embeddings

    —

    by

    Andrew
    in ,

    Search engines make use of AI to improve their search results. There are AI models that can understand the meaning of a sentence or document. They often present their results as embeddings of the document space into a vector space: D→ℝnD \to \mathbb{R}^n. These are called vector embeddings. Once you’ve found the embeddings for your…

  • Golden SSTables

    —

    by

    Andrew
    in

    When you’re testing, it’s common to transform whatever structure you’re dealing with into text and compare that text to a fixed string of expected text. This is convenient because you get a nice visual representation of what you’re expecting the output to be. This approach can cause problems if the text representation doesn’t capture everything…

  • Cookies and GDPR

    —

    by

    Andrew
    in ,

    I’m trying to navigate the intricacies of privacy and building a search engine. This means understanding important privacy legislation, like GDPR, and what it means for collection of the data that’s needed to tune a search engine. Session Cookies. The classic approach to building a search engine is to give users session cookies, log their…

  • Generating User Data

    —

    by

    Andrew
    in ,

    A search engine depends on a feedback loop of users making queries, following links, returning to the search page and rewriting their queries. All these feed into an understanding of whether they’re finding the results they’re looking for. Bootstrapping a system like this is difficult because you don’t start with any users and your search…

  • Search Indexes and Memory

    —

    by

    Andrew
    in ,

    I’ve been working on v0 of the search engine which requires building a search index in the form of “posting lists”. I’ve build a pipeline that reads wikipedia documents, outputs all the words it finds and emits key-value pairs of (word, url). Then we group by word so we can lookup all documents that word…

  • Writing SSTables with Beam

    —

    by

    Andrew
    in , ,

    Apache Beam is an open source system for processing large datasets. It has both a realtime and a batch processing mode. The batch processing mode is based on Google’s internal Flume framework which I had the pleasure of using for 7 years while processing Android telemetry. It’s also the perfect system for building a search…

  • SSTable on SSD

    —

    by

    Andrew
    in

    When I first put the SSTable server on Google’s Compute Engine, I opted for the cheapest machine: an e2-small (2 cores and 2GB ram) with a standard persistent disk of 20 GiB. When I ran the server, each request was taking hundreds of milliseconds. Installed iotop and found I was getting at most 8 MiB/s…

  • Cloud Engine and SSL

    —

    by

    Andrew
    in

    Yesterday, I spent a lot of time setting up byte64.com, particularly trying to setup HTTPS and connecting it to a Google Compute Engine Machine. I learned lots and suffered just as much. I started with a single E2 compute engine VM and had Google Cloud install a wordpress service on it. From there, I looked…

  • Building an SSTable

    —

    by

    Andrew
    in ,

    SSTables are a critical piece of technology that holds up the modern web. It’s the basis for most modern databases, search backends and many other technologies. What it provides is a reasonably fast way to perform lookups in large datasets. SSTables are sorted string tables meaning both our lookup keys and the resulting values are…