Taming Context Windows: Hierarchical Memory Assembly and Token Budgeting for Agentic Coding
Large Language Models boast expanding context windows, yet shoving an entire repository into a prompt causes severe attention dilution, hallucinations, and catastrophic latency. Learn how to architect a four-tier hierarchical memory assembly pipeline that feeds LLMs exactly what they need.
1. The Needle-in-a-Haystack Fallacy in Codebases
While modern foundation models claim context windows of 128k to 1M tokens, research consistently reveals the 'Lost in the Middle' phenomenon: retrieval accuracy degrades sharply when critical function signatures are buried deep within thousands of lines of irrelevant boilerplate.
Effective agentic coding requires selective memory layering rather than indiscriminate ingestion.
2. The Four-Tier Memory Architecture
In our production workbench, context is assembled dynamically from four distinct tiers:
| Memory Layer | Persistence Scope | Typical Allocation | Content Stored |
|---|---|---|---|
| L1: Immediate Working Scratchpad | Active Step | 15% (~4,000 tokens) | Current cursor focus, 50 preceding lines, active diff chunk. |
| L2: Task Context & Thread Anchor | Current Task | 35% (~10,000 tokens) | User instruction, operator transaction log, failing test traces. |
| L3: AST Semantic Graph | Workspace Session | 30% (~8,000 tokens) | Exported type definitions, interface contracts, symbol dependency maps. |
| L4: Long-Term Epistemic Vault | Global Repository | 20% (~6,000 tokens) | Architecture rules, coding guidelines, security constraints, historic commits. |
3. AST-Driven Symbol Hoisting Algorithm
Instead of passing raw source files to the LLM, we parse target dependencies using Tree-Sitter and extract only the type declarations and function signatures, stripping away implementation bodies.
import tree_sitter_python as tspython
from tree_sitter import Language, Parser
PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)
def hoist_signatures_only(source_code: str) -> str:
# Extract function and class signatures, discarding implementation bodies
tree = parser.parse(bytes(source_code, "utf8"))
hoisted_signatures = []
for node in tree.root_node.children:
if node.type in ("function_definition", "class_definition"):
lines = source_code.splitlines()
start_line = node.start_point[0]
hoisted_signatures.append(lines[start_line].strip() + " ...")
return "\n".join(hoisted_signatures)
This simple transformation reduces dependency token footprints by 82% while preserving complete type and symbol visibility for the AI compiler.