89 lines
3.5 KiB
Markdown
89 lines
3.5 KiB
Markdown
# Architecture
|
|
|
|
## Logical architecture
|
|
|
|
```text
|
|
Web-Access/Data-Store MCP
|
|
|
|
|
| JSONL file or PostgreSQL source table
|
|
v
|
|
StageContext + numbered stages
|
|
|
|
|
+--> PostgreSQL checkpoint tables
|
|
+--> PgVector retrieval_items + HNSW index
|
|
+--> structured JSON logs and artifacts
|
|
|
|
|
+--> OpenAI-compatible chat/embedding provider
|
|
|
|
BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages
|
|
|
|
|
+--> query/answer stages
|
|
```
|
|
|
|
See the editable [C4 context diagram](./diagrams/c4-context.drawio) and
|
|
[C4 container diagram](./diagrams/c4-container.drawio) for the system boundary.
|
|
|
|
## Stage contracts
|
|
|
|
| Stage | Reads | Writes | Result |
|
|
|---:|---|---|---|
|
|
| 1 | JSONL or source table | `raw_documents`, `documents` | normalized documents |
|
|
| 2 | documents | `chunks` | cleaned, deduplicated chunks |
|
|
| 3 | settings/model file | `knowledge_models` | entity and relationship vocabulary |
|
|
| 4 | chunks, model | `extractions` | validated LLM extraction payloads |
|
|
| 5 | extractions | `resolved_entities`, `resolved_relationships` | canonical entities and links |
|
|
| 6 | resolved data | `nodes`, `edges` | graph materialization |
|
|
| 7 | graph | `communities` | detected communities and LLM summaries |
|
|
| 8 | chunks, graph, communities | `search_items` | embeddings and HNSW index |
|
|
| 9 | search index, graph, documents | none | vector context, graph context, sources |
|
|
| 10 | stage 9 result | none | cited answer text and sources |
|
|
|
|
Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds
|
|
the question, retrieves `top_k` items, expands graph neighbors for the
|
|
configured number of hops and limit, and resolves source URLs.
|
|
|
|
## Durable execution
|
|
|
|
`pipeline_runs` and `stage_runs` are separate from stage data. A run can be:
|
|
|
|
```text
|
|
queued -> running -> succeeded
|
|
\-> failed -> queued (retry)
|
|
```
|
|
|
|
The worker:
|
|
|
|
1. initializes and verifies the schema;
|
|
2. requeues stale runs older than `WORKER_STALE_RUN_SECONDS`;
|
|
3. claims one queued run using database locking;
|
|
4. takes a per-corpus PostgreSQL advisory lock;
|
|
5. executes the selected stage range;
|
|
6. persists stage status and outputs;
|
|
7. releases the lock.
|
|
|
|
If a corpus lock is unavailable, execution is deferred and the job remains
|
|
queued. Retry resumes after the last successful stage unless the caller uses
|
|
`--reset` or otherwise clears downstream state.
|
|
|
|
## Deployment topology
|
|
|
|
`compose.yaml` starts PostgreSQL, waits for its health check, runs the one-shot
|
|
`migrate` service, then starts `api` and `worker`. The API and worker share the
|
|
application environment and bind-mount `./artifacts` into `/app/artifacts`.
|
|
|
|
The [C4 container diagram](./diagrams/c4-container.drawio) shows these
|
|
containers and external dependencies. The [component diagram](./diagrams/c4-component.drawio)
|
|
maps the application package, and the [code diagram](./diagrams/c4-code.drawio)
|
|
maps the numbered stage pipeline.
|
|
|
|
## Failure and consistency model
|
|
|
|
- Source records are rejected before they become normalized documents if their
|
|
schema, required fields, or corpus are invalid.
|
|
- A completed stage is a checkpoint, not a transaction spanning the whole build.
|
|
- A failed stage may leave partial stage side effects; reset the affected stage
|
|
and downstream state when a clean rebuild is required.
|
|
- PostgreSQL is required to exercise PgVector and HNSW behavior.
|
|
- Embedding vector length must equal `LLM_EMBEDDING_DIMENSIONS`.
|
|
|