I suppose I could implement PageRank on Wikipedia, but I figure that most Wikipedia pages are going to be high quality and what matters most is the relevance of a result to the search query. So, instead I’ve start generating some user data using using AI.<\/p>\n\n\n\n
I’m calling my tool a prober. It generates a query intent, translates that into a short query string, sees the results given back by the search system and then evaluates the top five results to see whether they match its original intent. It is currently looking at the first 1000 bytes of markdown for each result, but I’m going to have it change to look at the whole page if it’s not too expensive.<\/p>\n\n\n\n
Here’s an example prober result:<\/p>\n\n\n\n You can find more examples here:<\/p>\n\n\n\n https:\/\/byte64.com\/search\/#prober<\/a><\/p>\n\n\n\n You can see the intent which is separate from the query. The intent is used to judge each of the results. For this query, we got quite lucky that the main python pages were in the top five results. You’ll notice that there’s a warning about “language”. I’ve excluded that from my search index keys because it appears on every single wikipedia page. I’m going to need a more careful parser to include these words when they’re important to the page (e.g. https:\/\/en.wikipedia.org\/wiki\/English_language<\/a>) and skip them only when they’re not relevant.<\/p>\n\n\n\n It’s quite a new development to have AI in these parts of search. AI has been used for a long time to tune various parts of the scoring function, but it’s new to have it judge relevance or quality. Typically, to evaluate the quality of results, search companies outsource the work to human raters which are provided with a scoring guide. Those guides say out to rate relevance of a specific result to a query, whether one result is better than another, how authoritative a source is, etc. Now, we can have a LLM read a scoring guide and do this evaluation faster, cheaper and hopefully with comparable accuracy.<\/p>\n\n\n\n Here’s the basic requests I’m making to generate the intent and evaluate the relevance:<\/p>\n\n\n\n One thing I haven’t figured out yet is how to make the search intent generate something new each time. I have it running every 20 minutes and we’ll see how often it asks about the Apollo missions. I may end up randomly picking documents and ask it to form an intent and query for that document.<\/p>\n\n\n\n Where to go next? The results of the prober are provided by the prober itself, but a search engine should do its own logging to understand user behavior. Soon, I’ll build logs on the server side to see who’s querying, about what, and whether they’re getting their desired results.<\/p>\n\n\n\n ~Andrew<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" A search engine depends on a feedback loop of users making queries, following links, returning to the search page and rewriting their queries. All these feed into an understanding of whether they’re finding the results they’re looking for. Bootstrapping a system like this is difficult because you don’t start with any users and your search […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,18],"tags":[21,7],"class_list":["post-83","post","type-post","status-publish","format-standard","hentry","category-coding-with-ai","category-search","tag-llm","tag-wikipedia"],"_links":{"self":[{"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts\/83","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=83"}],"version-history":[{"count":4,"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts\/83\/revisions"}],"predecessor-version":[{"id":88,"href":"https:\/\/byte64.com\/index.php?rest_route=\/wp\/v2\/posts\/83\/revisions\/88"}],"wp:attachment":[{"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=83"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=83"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/byte64.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=83"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/byte64.com\/wp-content\/uploads\/2026\/03\/Screenshot-2026-03-18-at-2.01.17-p.m-1024x895.png\")
<\/figure>\n\n\n\nlet search_prompt = \"You are simulating a user browsing Wikipedia. Pick a random, specific topic (not too broad like 'history', but something like 'Apollo 11 mission' or 'Python programming language'). Write a brief sentence describing your search intent, and then provide a 2 to 4 word search query based on that intent.\";\n\nlet eval_prompt = format!(\n \"Original Search Intent: {}\\n\\n\"\n + \"Document Snippet (first 10000 chars):\\n---\\n{}\\n---\\n\\n\"\n + \"Please evaluate how well this document matches the user's intent. Provide a relevance score from 0.0 to 1.0 and a brief explanation.\",\n gemini_data.intent, snippet);<\/code><\/pre>\n\n\n\n