corpus runtime pipeline

corpus runtime pipeline An architecture diagram generated by Archify. cli · entrypoint (click) · Architecture component cli entrypoint (click) Gmail / IMAP · mail sources · Architecture component Gmail / IMAP mail sources api / mcp_server · REST + MCP tools · Architecture component api / mcp_server REST + MCP tools fetchers · build_fetcher → Record · Architecture component fetchers build_fetcher → Record classify · label + confidence · Architecture component classify label + confidence embeddings · Embedder (HTTP) · Architecture component embeddings Embedder (HTTP) store · Postgres / pgvector · Architecture component store Postgres / pgvector search · vector + SQL · Architecture component search vector + SQL enrichment · enrich_batch · enricher · Architecture component enrichment enrich_batch · enricher secret scan · scan · pii · leaks · Architecture component secret scan scan · pii · leaks OpenAI-compatible gateway · /v1 embeddings + chat · Architecture component OpenAI-compatible gateway /v1 embeddings + chat OTLP collector · traces + metrics · Architecture component OTLP collector traces + metrics IMAP / Gmail API Record labeled Record Document (vector + meta) HNSW kNN + SQL semantic_search / query POST /embeddings (batch) documents (cursor) documents (cursor) POST /chat/completions (guided JSON) ingest (batch) OTLP spans + metrics

Ingest line

  • • fetchers (IMAP / Gmail API) → classify → embeddings → pgvector store; ingest.py orchestrates, resumable via the sync_state cursor

Derived branches

  • • enrichment reads documents, calls the model (guided JSON), upserts a derived enrichments table in the same store
  • • secret scan = deterministic pii + leaks (Betterleaks); enrich_batch has the model confirm flagged candidates

Query + entrypoints

  • • search (HNSW + SQL) behind the api / mcp_server adapters; cli runs api, mcp, ingest, scan, enrich, audit-secrets