Corey James- 1,024dimensions per embedding
- ~19,000files in the monorepo's code index
- ~294,000structure-aware chunks embedded from them
A model behind a service boundary
The memory capability in FAFOHive searches code, reference documents and past decisions, and all of it depends on one multilingual embedding model. The first version ran that model in-process through a Python binding inside the Rust service. That put a Python runtime and a GPU model inside the Rust service, and every other service that needed embeddings would have loaded its own copy.
I moved the model behind a service boundary, so it is loaded once and shared. Hugging Face's Text Embeddings Inference server runs that model on a local GPU, and the memory service calls it over HTTP through a provider abstraction. I kept the service's existing embedding interface fixed while the backend changed underneath it, so no caller had to change.
the same interface, over HTTP
I make deliberate calls about where a model should live in a system and how a migration should land without breaking callers.
One interface, several backends
The provider abstraction keeps a CPU-only path, so the service still works on a machine without a GPU, and it can point at any OpenAI-compatible endpoint.
The same model serves the service's other search tools. The retrieval built on these embeddings is described under Hybrid retrieval, and the code-graph side of the service is A code explainer that is not allowed to invent.
Transport and capacity
The client authenticates to the inference server with mutual TLS, and the capacity layer treats the embedding endpoint as a bounded resource.
The inference server is Hugging Face's, used as provided; my work is the client, the migration, the transport security and the capacity behaviour around it.