Corey James- 3index families, each chosen for the set it held
- 7Bdense model in FP8 whose answers the pipeline searched
- 1bulk update per build writes index positions back
Searching for the gaps in what a model knew
Inference Fabric grew out of a research pipeline I ran first, and FAISS was the vector engine in both. The pipeline interrogated a model through vLLM, embedded the answers with an embedding model, grouped them into concept clusters of question-and-answer units, and searched for gaps in what the model knew. FAISS ran the batch nearest-neighbour search that found those gaps, rebuilt the GPU indexes after each restructuring pass, and fed the centroid updates.
The knowledge base reached hundreds of thousands of clusters and millions of units. I picked the index type per set, because the centroids, the answer embeddings and the candidates differ by orders of magnitude in size and in how exact the search needs to be. The builder pulls vectors out of PostgreSQL with the binary COPY protocol and parses the vector type's binary layout itself, which keeps a multi-million-vector load to a single sequential transfer.
set sizes illustrative
An index service, and where its cost sat
When the fabric needed an index service, I wrote the FAISS index manager as a function on my execution substrate. I staged vectors in an unlogged table, because the staging data is discarded after every build and gains nothing from write-ahead logging, and index positions go back to PostgreSQL through set-based stored procedures.
That design showed me where the cost sat: every vector made a round trip through SQL and a Python process. I replaced the FAISS path in the fabric once I decided vectors should stay on the GPU and out of SQL, and I kept the FAISS-computed distances that the health kernels still use.
- Materialise vectors from database viewsSQL
- Stage them for the buildunlogged table
- Build the index familiesPython subprocess
- Write index ids and offsets backstored procedures
- The staging tabledropped
FAISS carried a research pipeline to millions of vectors; when the fabric needed GPU-resident storage with its own guarantees, I built that and kept the one FAISS output that still paid for itself. I used FAISS as a library and did not modify it.
I use a library where it solves the problem at hand and replace it when the architecture outgrows it.
Index maintenance as an agent tool
FAFOHive gives agents the index manager as an MCP (Model Context Protocol) tool, beside the other maintenance jobs for the same knowledge base.