Technology · Model

Text embedding models

I ran a bake-off across embedding families for retrieval and replaced an incumbent chosen by habit with one chosen by measurement.

Capabilities model and FAFOHiveONNX Runtime and a local embedding serverOpen embedding models
Two rankings, one list
dense vectorslexical searchfused, then reranked
each bar a result, by score · illustrative
  • 5embedding models through one harness on the frozen evaluation
  • 1,024dimensions per vector in the memory index
  • 568Mparameters in the multilingual memory embedder

A bake-off instead of habit

Capabilities model needed a retrieval baseline strong enough that beating it would mean something. The incumbent embedding model had been picked only because the rest of the estate used it.

Estate consistency is no evidence.

I had a bake-off run on the frozen evaluation. The candidates went through the same harness on ONNX Runtime, and disqualifiers for latency and context length stopped an accurate but undeployable model from winning. The winner went into a fusion baseline with BM25. GPU scoring runs with TF32 pinned off, after a known-answer control showed that the precision setting changed rankings. The models are third-party open models; the bake-off, the serving architecture and the retrieval design are mine. See The baseline that tied us.

Candidates by size, millions of parameters
Sentence model22English retrieval model33General-text model30Long-context English model137Long-context, resizable vectors137
The fusion baseline against my fine-tuned model
recall@5, baselinerecall@5, fine-tunedrecall@10, baselinerecall@10, fine-tunedBuried requests, baselineBuried requests, fine-tuned
ties at 5 and 10; ahead on buried requests (lengths illustrative)
I build the strongest honest baseline even when it ties my own model.

The memory embedder as its own server

The memory capability in FAFOHive embeds code, documents and observations with a multilingual model that has dense and sparse outputs, on a local text-embedding inference server on my own GPU. The first version ran the model in-process through a Python bridge. I moved it to its own server behind an HTTP client, which took the Python runtime out of the Rust service.

Dense retrieval there always pairs with lexical search and a reranker, because similarity misses exact identifiers. See A code explainer that is not allowed to invent.

First version
Rust service
Python bridge
Embedding modelloaded in-process
Its own server
Rust service
HTTP clientno Python runtime
Embedding server
Embedding modelits own process on a local GPU
The model became a component with its own lifecycle instead of a dependency inside the service.

Backfill as an ordinary job

FAFO Core and Function Forge runs embedding backfill as an ordinary function. A job finds schema backlog, prioritises it and spawns workers against the same kind of embedding service, so embedding a new corpus is a scheduled, controllable job with the same controls as any other workload.

Schema backlog, prioritised
Worker 1
Worker 2
Worker 3
the highest-priority backlog goes firstworkers against the embedding service