ostk-recall
An autonomous memory and code indexing engine. Connects as a standard Model Context Protocol (MCP) server to feed vector embeddings, search indexes, and relational event streams to your AI agent.
License: AGPL-3.0 | Interface: stdio (MCP) | Engine: LanceDB + Tantivy
OVERVIEW
ostk-recall watches your workspace directory in the background. It reads modified files, splits them into semantic code chunks, computes embeddings locally, and hosts a search engine. When your LLM client queries the codebase, ostk-recall supplies relevant context snippets, preventing prompt drift and memory loss.
Incremental File Watching
Instead of scanning the whole filesystem on every prompt, a background file watcher monitors updates. File events are debounced and trigger incremental scans via the Unix socket at corpus.root/recall.sock (defaulting to ~/.local/share/ostk-recall/recall.sock). Watch mode supports incremental (~250ms debounce) or legacy (full re-scan) cadences.
THE INDEXING & SEARCH PIPELINE
The search engine combines semantic vector similarity and full-text keyword indexing:
Vector Search (LanceDB)
Embeds chunks locally using the model2vec-rs library. The default model is minishlab/potion-retrieval-32M, producing dense 512-dimension vectors with zero network latency.
Full-Text & Reranking
Queries are tokenized and scored using Tantivy (BM25). The combined vector and text results are then re-scored locally using the fastembed-rs integration with jina-reranker-v1-turbo-en.
Local Vectorization & Symbol-Aware AST Chunking
ostk-recall avoids naive line-by-line or character-window chunking. It parses codebases syntax-first to preserve structural relationship context.
1. Local Vectorization Mechanics
Vector generation runs completely local via model2vec-rs using the minishlab/potion-retrieval-32M model (512-dimensional output).
- Zero Latency: Embeddings are computed in ~0.8ms per chunk on Apple Silicon GPUs (Metal backend) or standard CPU threads.
- Privacy: Code contents never leave localhost during indexing, resolving standard enterprise data-leak concerns.
2. Symbol-Aware AST Chunking via `fcp-rust`
Instead of cutting chunks mid-expression, ostk-recall employs tree-sitter based parsers (like fcp-rust for Rust):
- Syntactic Boundaries: Chunks are split at logical AST nodes—structs, impl blocks, functions, and enums.
- Context Parentage: Child nodes inherit the parent class/struct signature automatically, so that individual method chunks retain context of the class they belong to.
3. File Path Weight-Multiplication
To prioritize direct file navigation queries, paths are enriched during index registration:
- File path tokens (e.g.
src/kernel/cas.rs→src,kernel,cas) are appended to the document's keyword vector. - The indexing weights for path tokens are multiplied by a boost factor (default:
3.0x). This guarantees that queries mentioning a file path or module name immediately bring up the containing symbol definition chunks first.
EXPOSED MCP TOOLS
When registered as an MCP server in your client configuration, ostk-recall exposes the following specialized tools:
recall query: String Performs hybrid semantic and full-text search across the workspace index and logs. recall_link source: Path, target: Path Creates an explicit attribution edge in the semantic dependency graph. recall_stats — Returns vector storage sizes, file watch queues, and cache hit ratios. recall_audit sql: String Executes direct SQL SELECT queries over the SQLite audit_events table. recall_fault target_dir: Path Runs synthesizers to locate undocumented API gaps or architectural discrepancies (relay for haystack mem.fault_recall). CLIENT CONFIGURATION
To integrate memory into Claude Code or Cursor, register the server in your MCP settings:
{
"mcpServers": {
"ostk-recall": {
"command": "ostk-recall",
"args": ["serve", "--stdio"],
"env": {
"OSTK_WORKSPACE": "/Users/username/projects/my-workspace"
}
}
}
}