RAG-POC/docs/architecture.md

3.5 KiB

Architecture

Logical architecture

Web-Access/Data-Store MCP
          |
          | JSONL file or PostgreSQL source table
          v
StageContext + numbered stages
          |
          +--> PostgreSQL checkpoint tables
          +--> PgVector retrieval_items + HNSW index
          +--> structured JSON logs and artifacts
          |
          +--> OpenAI-compatible chat/embedding provider

BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages
                                      |
                                      +--> query/answer stages

See the editable C4 context diagram and C4 container diagram for the system boundary.

Stage contracts

Stage Reads Writes Result
1 JSONL or source table raw_documents, documents normalized documents
2 documents chunks cleaned, deduplicated chunks
3 settings/model file knowledge_models entity and relationship vocabulary
4 chunks, model extractions validated LLM extraction payloads
5 extractions resolved_entities, resolved_relationships canonical entities and links
6 resolved data nodes, edges graph materialization
7 graph communities detected communities and LLM summaries
8 chunks, graph, communities search_items embeddings and HNSW index
9 search index, graph, documents none vector context, graph context, sources
10 stage 9 result none cited answer text and sources

Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds the question, retrieves top_k items, expands graph neighbors for the configured number of hops and limit, and resolves source URLs.

Durable execution

pipeline_runs and stage_runs are separate from stage data. A run can be:

queued -> running -> succeeded
                 \-> failed -> queued (retry)

The worker:

  1. initializes and verifies the schema;
  2. requeues stale runs older than WORKER_STALE_RUN_SECONDS;
  3. claims one queued run using database locking;
  4. takes a per-corpus PostgreSQL advisory lock;
  5. executes the selected stage range;
  6. persists stage status and outputs;
  7. releases the lock.

If a corpus lock is unavailable, execution is deferred and the job remains queued. Retry resumes after the last successful stage unless the caller uses --reset or otherwise clears downstream state.

Deployment topology

compose.yaml starts PostgreSQL, waits for its health check, runs the one-shot migrate service, then starts api and worker. The API and worker share the application environment and bind-mount ./artifacts into /app/artifacts.

The C4 container diagram shows these containers and external dependencies. The component diagram maps the application package, and the code diagram maps the numbered stage pipeline.

Failure and consistency model

  • Source records are rejected before they become normalized documents if their schema, required fields, or corpus are invalid.
  • A completed stage is a checkpoint, not a transaction spanning the whole build.
  • A failed stage may leave partial stage side effects; reset the affected stage and downstream state when a clean rebuild is required.
  • PostgreSQL is required to exercise PgVector and HNSW behavior.
  • Embedding vector length must equal LLM_EMBEDDING_DIMENSIONS.