RAG-POC/docs/data-model.md

39 lines
1.5 KiB
Markdown

# Data model and checkpoint ownership
The initial Alembic revision creates the SQLAlchemy metadata and the PgVector
extension. The models live in [`src/graphrag_platform/models.py`](../src/graphrag_platform/models.py).
## Pipeline data
| Table/model | Owner stage | Meaning |
|---|---:|---|
| `raw_documents` | 1 | original source payload and provenance |
| `documents` | 1 | normalized text and source identity |
| `chunks` | 2 | cleaned, overlapping retrieval units |
| `knowledge_models` | 3 | configured entity/relationship vocabulary and templates |
| `extractions` | 4 | validated LLM payload per chunk |
| `resolved_entities` | 5 | canonical entities and aliases |
| `resolved_relationships` | 5 | relationships linked to canonical keys |
| `nodes` | 6 | graph nodes |
| `edges` | 6 | graph edges |
| `communities` | 7 | detected groups and summaries |
| `search_items` | 8 | text, metadata, source URL, and embedding |
## Run tracking
| Table/model | Purpose |
|---|---|
| `pipeline_runs` | requested range, status, attempt, caller, correlation, and errors |
| `stage_runs` | per-stage status, input/output, timestamps, and error details |
Stage data is corpus-scoped. Clear or rebuild from the first invalid stage
through downstream derived data; do not manually delete unrelated corpora.
## Indexing
PostgreSQL creates a dimension-specific HNSW cosine index on retrieval items.
SQLite stores vectors as JSON and uses in-process cosine similarity. The latter
is useful for tests but does not validate production query plans or PgVector
behavior.