I’m trying to navigate the intricacies of privacy and building a search engine. This means understanding important privacy legislation, like GDPR, and what it means for collection of the data that’s needed to tune a search engine.
Session Cookies. The classic approach to building a search engine is to give users session cookies, log their searches, clicks, query rewrites, etc. along with the session cookie, and then process the resulting logs to understand how to improve search results.
The problem with this approach in the recent legal environment is that the session cookies form an identity for the user and, while it’s essential to improving the search engine, it’s not essential to fulfilling the user’s request as best we can at the time. This means that users need to be asked for consent before giving them this kind of identifier and logging against it.
Client-side Sessions. There is another approach that’s worth exploring. The kinds of logs we need span short periods of time where a user is writing queries, looking at the results, refining their queries and so on. Instead of building up these sessions from information to a server, we can instead build those sessions on the client’s machine using local storage and send the entire session to the server when it’s done. This way, the server doesn’t need to keep any identifier to join the separate event records.
My understanding is that building client-side sessions is preferable from a regulatory standpoint. This seems odd to me since the information contained in the client-side session is exactly the same as the information in the logged events that share a randomly generated session id. So, I think I really need to understand if this approach is preferred. Likely, we’d need to compute the metrics we’re interested in on the client side, rather than simply bundle the events. However, this requires knowing all the analysis that you want to do in advance. Every time you want a new metric, you have to put it out in the field and wait to collect data before knowing if it’s valuable. That’s a lot harder than simply running queries over the month of logs you’ve collected.
Cookies on Byte64. Here’s how I’ve done the implementation so far. First, we need to provide users with a way of opting in to logging and, likewise, a way to turn it off. This user preference needs to be persistent — it’s frowned upon to ask the user for their consent over and over, every time they access your product. So, what we need is a preference cookie. On the backend, this will be associated to the user’s logging choice. The cookie will last for year of inactivity, so whenever a user comes back, we remember if they want to be part of session logging or not. This cookie isn’t considered a tracking or identifying cookie, because we don’t produce any logs against it. It’s never associated with your activity on the site.
Next we need the actual session cookie. A session cookie is simply a cookie without an expiry date. That seems like the opposite of what we want — we want a short lived cookie to tie together a few events that happen in quick succession. But, browsers won’t save a cookie without an expiry to disk. This means whenever you close your browser window they disappear. That’s cold comfort for someone like me who rarely ever closes their browser. So, in addition, I’ve added a server-side expiry of 30 minutes of inactivity. That means after 30 minutes of not using your cookie, if you try to use it again, we’ll issue a fresh one associated to a completely new server-side identifier. Now, if your logging preferences (judged based on your preference cookie) says you permit logging, then every time a server log gets written, it’ll include the server-side identifier that goes with your cookie.
Normally, users have no visibility into their cookies. You have to go digging to see all the cookies stored for a given site and see their expiration dates. Even then, you have no idea what’s being logged against these cookies. In an effort to be more transparent, I’ve exposed the last byte of your internal session id. Rather than showing the byte, I’ve picked 255 names — one for each value. This way, if the name associated to your account changes, you can tell that your session cookie has been rotated. You still have to take it on faith that your logging preferences are being followed. Every http request to byte64 will contain both your preference and session cookie and someone nefarious could track users across either.
One last fun design choice I’ve made is to do automatic cookie rotation. Whenever you send an http request your preference and session cookies will be sent along. When the response comes back, you’ll receive new values for each. In fact, these are really the same cookies because, on the server side, they’re being associated with a fixed internal id (separate for pref and session) that is not changing. The advantage of this is that http requests can’t be replayed because you can’t send the same cookie twice and have it associated with the same internal id. If you try that, you’ll be assigned a new internal id and be treated as a fresh user.
~Andrew
Leave a Reply