Corey James- 19,000files in the multi-language monorepo
- 294,000chunks from the monorepo
- 100lines, the longest unit kept whole
- 40lines per overlapping sub-chunk
Answers from the code's structure
FAFOHive serves AI agents that ask questions about a large multi-language monorepo, and I wanted their answers to come from the code's structure. Embeddings of arbitrary line windows give agents half a function and a guess. I decided to parse every file with tree-sitter and use the syntax tree twice: to cut code into meaningful chunks for search, and to extract the dependency graph the code tools traverse.
Chunks that follow syntax
The chunker uses pinned grammars for Rust, Python, JavaScript, TypeScript, C, C++, Go and Java and cuts along function, class and module boundaries. A long unit splits into overlapping sub-chunks, so a long function stays findable without becoming one diluted vector. Files in other languages go through a plain chunker.
Every chunk carries its path, line range and language, so every hit points at a file and line. Grammar versions are pinned, which keeps chunk boundaries and edge identities stable between runs; I first built this in a small standalone code-search server, and the index pins the same grammar versions so boundaries stayed stable when the work moved.
A graph that is never half built
The graph extractor re-parses those chunks and extracts calls, imports, implementations, references, definitions and re-exports. An edge waits in a pending state until both of its ends are indexed, and every edge belongs to a graph generation, so I can rebuild an index while queries read the previous complete graph. Graph coverage on the monorepo reached all but a few hundred chunks.
A model that may only explain
Dependency tracing is a bounded breadth-first search in SQL with no model in it, and path explanation lets a language model narrate only edges the graph already holds. When coverage for a query is incomplete, the explainer says "coverage incomplete". That design is A code explainer that is not allowed to invent. tree-sitter and its grammars are used as published; the chunking strategy, the edge model and the graph lifecycle are mine.
I give AI agents computed facts about code and limit the model to explaining them.