{"id":102,"date":"2026-03-31T04:11:12","date_gmt":"2026-03-31T04:11:12","guid":{"rendered":"https:\/\/byte64.com\/?p=102"},"modified":"2026-03-31T14:44:36","modified_gmt":"2026-03-31T14:44:36","slug":"vector-retrieval","status":"publish","type":"post","link":"https:\/\/byte64.com\/?p=102","title":{"rendered":"Vector Retrieval"},"content":{"rendered":"\n

Now that every document has been assigned a vector encoding its semantics, this opens the door to a new kind of retrieval. Rather than find documents that might be relevant to the query by searching through the search index for keywords, we can instead take the query’s vector and find nearby documents in the embedding. <\/p>\n\n\n\n

One advantage this has over keyword retrieval is that we know that all the documents we retrieve this way are on the right topic. So rather than going through tens of thousands of documents and scoring them, they come pre-scored by the embedding and so we need a lot fewer of them. Right now, the byte64 search engine is setup to pull only the 50 nearest neighbour documents to the query’s vector. And the results are great.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Let’s talk about the infrastructure that supports this kind of retrieval. The fundamental problem of exact vector-based nearest neighbour retrieval<\/a> is roughly linear in the worst case. Approximate methods can speed this up considerably. Off the shelf, you can get Google’s Scaleable Nearest Neighbor library<\/a> which provides everything you need — building an index and serving that’s optimized for x86 architecture. It can pull the nearest 50 documents in under 30ms. But for the wikipedia dataset, it costs another 4 GiB of ram.<\/p>\n\n\n\n

I tried rolling my own approximate nearest neighbour algorithm. My idea was do divide the vectors in \ud835\udd4a<\/mi>127<\/mn><\/msup>=<\/mo>{<\/mo>v<\/mi>\u2208<\/mo>\u211d<\/mi>128<\/mn><\/msup>|<\/mo>\u2016<\/mi>v<\/mi>\u2016<\/mi>=<\/mo>1<\/mn>}<\/mo><\/mrow>\\mathbb{S}^{127} = \\{ v \\in \\mathbb{R}^{128} \\mid \\|v\\| = 1\\}<\/annotation><\/semantics><\/math> into different cells C<\/mi>i<\/mi>j<\/mi><\/mrow><\/msub>C_{ij}<\/annotation><\/semantics><\/math>. Then given a query vector q<\/mi>\u2208<\/mo>\ud835\udd4a<\/mi>127<\/mn><\/msup><\/mrow>q \\in \\mathbb{S}^{127}<\/annotation><\/semantics><\/math> we’ll select some cells that we think the candidates lie in. If the vectors within the cells are nicely ordered, we can even merge them using using a heap until we get the number of documents we want.<\/p>\n\n\n\n

Given a vector v<\/mi>=<\/mo>(<\/mo>v<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>v<\/mi>128<\/mn><\/msub>)<\/mo><\/mrow>v = (v_1, \\ldots, v_{128})<\/annotation><\/semantics><\/math> let v<\/mi>i<\/mi><\/msub>v_i<\/annotation><\/semantics><\/math> and v<\/mi>j<\/mi><\/msub>v_j<\/annotation><\/semantics><\/math> be the largest and second largest component of v (taken in absolute values). We define C<\/mi>i<\/mi>j<\/mi><\/mrow><\/msub>C_{ij}<\/annotation><\/semantics><\/math> to be the collection of all such vectors — actually we need to account for the signs of v<\/mi>i<\/mi><\/msub>v_i<\/annotation><\/semantics><\/math> and v<\/mi>j<\/mi><\/msub>v_j<\/annotation><\/semantics><\/math> so we really need C<\/mi>i<\/mi>j<\/mi><\/mrow>+<\/mo>+<\/mo><\/mrow><\/msubsup>,<\/mo>C<\/mi>i<\/mi>j<\/mi><\/mrow>\u2212<\/mo>+<\/mo><\/mrow><\/msubsup>,<\/mo>C<\/mi>i<\/mi>j<\/mi><\/mrow>+<\/mo>\u2212<\/mo><\/mrow><\/msubsup>,<\/mo>C<\/mi>i<\/mi>j<\/mi><\/mrow>\u2212<\/mo>\u2212<\/mo><\/mrow><\/msubsup><\/mrow>C_{ij}^{++}, C_{ij}^{-+}, C_{ij}^{+-}, C_{ij}^{–}<\/annotation><\/semantics><\/math>. So, this defines 4<\/mn>\u22c5<\/mo>128<\/mn>\u22c5<\/mo>127<\/mn><\/mrow>4 \\cdot 128 \\cdot 127<\/annotation><\/semantics><\/math> different cells. <\/p>\n\n\n\n

When we’re trying to find vectors near q<\/mi>=<\/mo>(<\/mo>q<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>q<\/mi>128<\/mn><\/msub>)<\/mo><\/mrow>q = (q_1, \\ldots, q_{128})<\/annotation><\/semantics><\/math>, we sort the components in absolute values as |<\/mi>q<\/mi>\u03c3<\/mi>(<\/mo>1<\/mn>)<\/mo><\/mrow><\/msub>|<\/mi>\u2265<\/mo>|<\/mi>q<\/mi>\u03c3<\/mi>(<\/mo>2<\/mn>)<\/mo><\/mrow><\/msub>|<\/mi>\u2265<\/mo>\u22ef<\/mo>\u2265<\/mo>|<\/mi>q<\/mi>\u03c3<\/mi>(<\/mo>128<\/mn>)<\/mo><\/mrow><\/msub>|<\/mi><\/mrow>|q_{\\sigma(1)}| \\geq |q_{\\sigma(2)}| \\geq \\cdots \\geq |q_{\\sigma(128)}| <\/annotation><\/semantics><\/math> and then search in C<\/mi>\u03c3<\/mi>(<\/mo>1<\/mn>)<\/mo>\u03c3<\/mi>(<\/mo>2<\/mn>)<\/mo><\/mrow>s<\/mi>1<\/mn><\/msub>s<\/mi>2<\/mn><\/msub><\/mrow><\/msubsup>,<\/mo>C<\/mi>\u03c3<\/mi>(<\/mo>1<\/mn>)<\/mo>\u03c3<\/mi>(<\/mo>3<\/mn>)<\/mo><\/mrow>s<\/mi>1<\/mn><\/msub>s<\/mi>3<\/mn><\/msub><\/mrow><\/msubsup>,<\/mo>C<\/mi>\u03c3<\/mi>(<\/mo>1<\/mn>)<\/mo>\u03c3<\/mi>(<\/mo>4<\/mn>)<\/mo><\/mrow>s<\/mi>1<\/mn><\/msub>s<\/mi>4<\/mn><\/msub><\/mrow><\/msubsup>,<\/mo>\u2026<\/mo><\/mrow>C_{\\sigma(1)\\sigma(2)}^{s_1 s_2}, C_{\\sigma(1)\\sigma(3)}^{s_1 s_3}, C_{\\sigma(1)\\sigma(4)}^{s_1 s_4}, \\ldots<\/annotation><\/semantics><\/math>in that order. Here s<\/mi>i<\/mi><\/msub>s_i<\/annotation><\/semantics><\/math> is the sign of q<\/mi>\u03c3<\/mi>(<\/mo>i<\/mi>)<\/mo><\/mrow><\/msub>q_{\\sigma(i)}<\/annotation><\/semantics><\/math>. <\/p>\n\n\n\n

I sorted the elements in C<\/mi>i<\/mi>j<\/mi><\/mrow><\/msub>C_{ij}<\/annotation><\/semantics><\/math> by their i-th component followed by j-th component, I was hoping next nearest neighbour would appear in C<\/mi>\u03c3<\/mi>(<\/mo>1<\/mn>)<\/mo>\u03c3<\/mi>(<\/mo>k<\/mi>)<\/mo><\/mrow>s<\/mi>1<\/mn><\/msub>s<\/mi>k<\/mi><\/msub><\/mrow><\/msubsup>C_{\\sigma(1)\\sigma(k)}^{s_1 s_k}<\/annotation><\/semantics><\/math> before C<\/mi>\u03c3<\/mi>(<\/mo>1<\/mn>)<\/mo>\u03c3<\/mi>(<\/mo>k<\/mi>+<\/mo>1<\/mn>)<\/mo><\/mrow>s<\/mi>1<\/mn><\/msub>s<\/mi>k<\/mi>+<\/mo>1<\/mn><\/mrow><\/msub><\/mrow><\/msubsup>C_{\\sigma(1)\\sigma(k+1)}^{s_1 s_{k+1}}<\/annotation><\/semantics><\/math>. But unfortunately, the largest component isn’t the best indication of where the largest dot-product will be. <\/p>\n\n\n\n

Take a look at the distribution of cosine similarities with an arbitrary nut-based wikipedia page.<\/p>\n\n\n\n

Cosine Similarity Distribution (against page: https:\/\/en.wikipedia.org\/wiki\/Pistachio):\nBucket Range         | Count     \n---------------------|-----------\n[-1.00, -0.90) | 0\n[-0.90, -0.80) | 0\n[-0.80, -0.70) | 0\n[-0.70, -0.60) | 0\n[-0.60, -0.50) | 0\n[-0.50, -0.40) | 0\n[-0.40, -0.30) | 22\n[-0.30, -0.20) | 2263\n[-0.20, -0.10) | 49199\n[-0.10,  0.00) | 343893\n[ 0.00,  0.10) | 1018456\n[ 0.10,  0.20) | 1997144\n[ 0.20,  0.30) | 2221347\n[ 0.30,  0.40) | 955750\n[ 0.40,  0.50) | 318725\n[ 0.50,  0.60) | 204906\n[ 0.60,  0.70) | 18468\n[ 0.70,  0.80) | 96\n[ 0.80,  0.90) | 0\n[ 0.90,  1.00) | 1<\/code><\/pre>\n\n\n\n
But if we use the approach above to retrieve the 10000 nearest neighbors of the document, we get this distribution:<\/p>\n\n\n\n
Similarity Distribution of top 10000 results:\nBucket Range         | Count     \n---------------------|-----------\n[-1.00, -0.90) | 0\n[-0.90, -0.80) | 0\n[-0.80, -0.70) | 0\n[-0.70, -0.60) | 0\n[-0.60, -0.50) | 0\n[-0.50, -0.40) | 0\n[-0.40, -0.30) | 0\n[-0.30, -0.20) | 0\n[-0.20, -0.10) | 0\n[-0.10,  0.00) | 0\n[ 0.00,  0.10) | 0\n[ 0.10,  0.20) | 0\n[ 0.20,  0.30) | 0\n[ 0.30,  0.40) | 0\n[ 0.40,  0.50) | 0\n[ 0.50,  0.60) | 5757\n[ 0.60,  0.70) | 4203\n[ 0.70,  0.80) | 39\n[ 0.80,  0.90) | 0\n[ 0.90,  1.00) | 1<\/code><\/pre>\n\n\n\n
So, we didn’t even retrieve half of the documents in the 0.7 to 0.8 range and those are all within the 100 nearest neighbours. This is pretty crumby recall, so I’m sticking with ScaNN for now.<\/p>\n\n\n\n
Two downsides to ScaNN is that it uses a python frontend to a C++ backend and it also requires x86. This complicates testing, since it requires docker on Mac M and it means python is needed on the search server. These kind of algorithms are incredibly fun to learn and write. Since it’s open source, I’d love to port it to rust if I can.<\/p>\n\n\n\n
~Andrew<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Now that every document has been assigned a vector encoding its semantics, this opens the door to a new kind of retrieval. Rather than find documents that might be relevant to the query by searching through the search index for keywords, we can instead take the query’s vector and find nearby documents in the embedding. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,10,18],"tags":[25],"class_list":["post-102","post","type-post","status-publish","format-standard","hentry","category-data-structures","category-google-cloud","category-search","tag-vector-embeddings"],"_links":{"self":[{"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts\/102","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=102"}],"version-history":[{"count":3,"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts\/102\/revisions"}],"predecessor-version":[{"id":106,"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts\/102\/revisions\/106"}],"wp:attachment":[{"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=102"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=102"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=102"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}