Suppose a help bot needs the passage about recovering a cancelled export. It searches the cluster of export setup documents, and the relevant recovery passage is in a different cluster. The language model can only work with the passages it receives. To debug that answer, start by finding out whether the retriever ever looked in the right place.
Work through four points
In this illustrative example the query is q = (1, 0), so a vector's score is simply its first coordinate. Partition A has (3, 1) and (3, −1). Partition B has (5, 0) and (−1, 0). The centroids score 3 and 2. A router with a one-partition budget opens A, but the global best key scores 5 and lives in B. You can reproduce all four scores by hand.
The diagram is an original Boloop illustration of this toy example. It depicts inner-product scores, with no benchmark data.
What Apple changed
In their research note, Apple describes two models trained on queries with exact best matches computed offline. SupportNet estimates how high a score each partition could contain. KeyNet predicts a key vector and uses that prediction to query an existing approximate index. Either can still miss the right match. The ICML paper is by Theo X. Olausson, João Monteiro, Michal Klein and Marco Cuturi.
On Apple's Natural Questions experiment, the small SupportNet reached 95% routing recall at 1.74 ms versus 3.36 ms for centroid routing. Routing recall here asks whether the system opens the partition holding the exact best key. The timing is for that experiment and includes the router's inference cost. It gives no estimate of end-to-end response time or answer accuracy for your corpus.
A practical evaluation
- Freeze a corpus snapshot and the embedding model. Save a held-out query set, including short queries, rare terms and questions about newly added documents. Keep the exact top-k from an exhaustive search over that same snapshot.
- Compare both routers at a matched recall target. Record routing recall, exact-neighbor recall@k, and p50/p95 retrieval latency under the same hardware and concurrency. A representative set of human-judged relevant passages can reveal cases where the nearest vector is a poor answer source.
- Decide what counts as failure before comparing results. Track each rare or fresh-query slice. If one loses important passages, try probing extra partitions for that slice and measure the added cost. Keep the old route available while validating changes in shadow traffic.
- Count retraining and index-refresh work. A stable corpus with many similar queries is a better candidate for amortizing that cost. If embeddings or documents change often, measure the gain again on a newer snapshot before rollout.
Our recommendation is to keep an exact-neighbor control and a relevance check even if a learned router looks faster. That makes it possible to tell which stage lost the evidence when an answer fails. For the broader habit of checking a real outcome, see the AI-generated code acceptance checklist.
Boloop has checked the sources and the toy example. We have not rerun Apple's benchmark.