RAG-POC/README.md

646 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# KnowledgeGraph GraphRAG
An internal, restartable GraphRAG pipeline that normalizes externally collected
content, extracts and reconciles knowledge, builds a graph, indexes graph and source
evidence with PostgreSQL/PgVector, retrieves graph-aware context, and generates
source-cited answers.
The collection process is outside this project. Existing **Web-Access MCP** and
**Data-Store MCP** processes discover and persist source material. This project
starts by validating and normalizing their versioned handoff.
## Architecture
```text
Web-Access MCP + Data-Store MCP
|
| versioned JSONL or PostgreSQL handoff
v
01 normalize and parse
v
02 clean, deduplicate, and chunk
v
03 define knowledge model
v
04 extract entities and relationships
v
05 reconcile aliases and duplicates
v
06 materialize graph nodes and edges
v
07 detect and summarize communities
v
08 build PgVector HNSW retrieval index
v
09 retrieve vector + graph + source context
v
10 generate a cited answer
```
PostgreSQL is the canonical checkpoint store. Every completed build stage persists
its output, so later stages can be rebuilt without repeating valid upstream work.
PgVector stores embeddings and performs server-side HNSW cosine search. Indexed
relational edge tables provide bounded graph expansion.
The deployment consists of:
- PostgreSQL 16 with PgVector
- Alembic migration job
- Internal FastAPI service
- PostgreSQL-backed asynchronous worker
- Existing BFF as the policy enforcement point
The API is internal only and is not intended for direct customer access.
## Repository layout
| Path | Purpose |
|---|---|
| [`src/01_normalize_content.py`](src/01_normalize_content.py) through [`src/10_generate_answer.py`](src/10_generate_answer.py) | Complete numbered stage implementations |
| [`src/graphrag_platform`](src/graphrag_platform) | Configuration, models, storage, orchestration, API, auth, worker, and provider clients |
| [`migrations`](migrations) | Alembic database migrations |
| [`tests`](tests) | Unit, API/worker, pipeline, migration, and real PgVector integration tests |
| [`compose.yaml`](compose.yaml) | Reproducible PostgreSQL, migration, API, and worker environment |
| [`Containerfile`](Containerfile) | Non-root application image |
| [`.env.example`](.env.example) | Runtime configuration template |
Detailed architecture, API, runbook, configuration, data model, testing, and
editable C4 diagrams are available in [`docs/`](docs/README.md).
## Prerequisites
- Python 3.11 or newer
- Podman 5 or newer
- `podman machine` running on Windows
- An OpenAI-compatible chat and embeddings endpoint
- External Web-Access MCP/Data-Store MCP collection process
## Quick start with Compose
Create the local environment:
```powershell
Copy-Item .env.example .env
```
Edit `.env` and set at minimum:
```dotenv
POSTGRES_PASSWORD=choose-a-local-password
DATABASE_URL=postgresql+psycopg://knowledgegraph_app:choose-a-local-password@localhost:5432/knowledgegraph
LLM_BASE_URL=https://api.openai.com/v1
LLM_API_KEY=replace-with-provider-key
LLM_CHAT_MODEL=gpt-4.1-mini
LLM_EMBEDDING_MODEL=text-embedding-3-small
LLM_EMBEDDING_DIMENSIONS=1536
INTERNAL_JWT_SECRET=replace-with-at-least-32-random-bytes
```
`POSTGRES_PASSWORD` and the password in `DATABASE_URL` must match. Compose replaces
the database host with `postgres` inside application containers.
Start the complete environment:
```powershell
podman machine start
podman compose up --detach --build
```
Compose:
1. Starts `docker.io/pgvector/pgvector:pg16`.
2. Waits for PostgreSQL readiness.
3. Runs `alembic upgrade head`.
4. Starts the API and worker.
5. Waits for API dependency readiness.
Verify it:
```powershell
podman compose ps
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready
```
View logs:
```powershell
podman compose logs --follow api worker
```
Stop services while retaining PostgreSQL data:
```powershell
podman compose down
```
Delete services and all local database data:
```powershell
podman compose down --volumes
```
Volume removal is destructive.
## External source handoff
### JSON Lines contract
Each UTF-8 JSONL line must satisfy schema version `1.0`:
```json
{
"schema_version": "1.0",
"external_id": "stable-source-record-id",
"corpus": "product-knowledge",
"url": "https://docs.example.com/article",
"body": "<html><body><main>Article text</main></body></html>",
"content_type": "text/html",
"status_code": 200,
"retrieved_at": "2026-10-13T18:00:00Z",
"metadata": {
"collector": "Web-Access",
"stored_by": "Data-Store"
}
}
```
Required fields:
- `schema_version`: currently `1.0`
- `external_id`: stable and unique within a corpus
- `url`
- `body`
- `retrieved_at`: ISO 8601 timestamp with offset
Optional/defaulted fields:
- `corpus`: when provided, it must match the selected runtime corpus
- `content_type`: defaults to `text/html`
- `status_code`: defaults to `200`
- `metadata`: defaults to an empty object
Unknown fields and unsupported schema versions are rejected. Invalid records include
the file and line number in the error.
The template is [`src/templates/mcp_documents.jsonl`](src/templates/mcp_documents.jsonl).
A larger acceptance corpus is
[`tests/fixtures/external_documents.jsonl`](tests/fixtures/external_documents.jsonl).
### PostgreSQL source contract
Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to read the same contract from an
external PostgreSQL table:
```dotenv
SOURCE_DATABASE_URL=postgresql+psycopg://source_reader:password@source-host:5432/source_database
SOURCE_TABLE_NAME=external_documents
```
Compatible table:
```sql
CREATE TABLE external_documents (
external_id text PRIMARY KEY,
schema_version text NOT NULL DEFAULT '1.0',
corpus text,
url text NOT NULL,
body text NOT NULL,
content_type text NOT NULL DEFAULT 'text/html',
status_code integer NOT NULL DEFAULT 200,
retrieved_at timestamptz NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);
```
Additional source-table columns are ignored. The application account should receive
read-only access to the source table.
Provide either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both.
## Durable checkpoints
Build stages persist these boundaries:
```text
raw_documents + documents
-> chunks
-> knowledge_models
-> extractions
-> resolved_entities + resolved_relationships
-> nodes + edges
-> communities
-> retrieval_items
```
`pipeline_runs` and `stage_runs` separately record:
- queued/running/succeeded/failed state
- selected stage range
- attempt
- configuration fingerprint
- caller and correlation ID
- stage inputs and outputs
- timestamps and error details
The worker requeues runs abandoned beyond `WORKER_STALE_RUN_SECONDS`. Set this value
above the expected maximum duration of one stage.
PostgreSQL advisory locks allow only one active build for a corpus. If another
worker claims a job for that corpus, the job is returned to the queue instead of
executing conflicting writes.
DataFrames are not stage checkpoints. A Pandas or Polars DataFrame may be used by a
future adapter for one bounded transformation, but every completed result must be
persisted before the next stage starts.
## Database migrations
Runtime services verify the migrated schema and do not silently mutate it.
Apply migrations:
```powershell
alembic upgrade head
```
Inspect the current revision:
```powershell
alembic current
alembic history
```
Create a migration after changing SQLAlchemy models:
```powershell
alembic revision --autogenerate -m "describe schema change"
```
Review generated migrations before applying them. Compose runs upgrades through the
one-shot `migrate` service before starting API and worker services.
`AUTO_CREATE_SCHEMA=true` exists only for isolated tests. Do not enable it in shared
dev, test, or production environments.
## Run stages directly
Install a local editable environment:
```powershell
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[test]"
```
Run migrations before the first command:
```powershell
alembic upgrade head
```
### 1. Normalize external content
```powershell
python .\src\01_normalize_content.py `
--corpus product-knowledge `
--input-jsonl .\artifacts\mcp-documents.jsonl
```
This validates the external handoff, stores raw provenance, runs Trafilatura for
HTML, and persists normalized document records.
### 2. Clean, deduplicate, and chunk
```powershell
python .\src\02_clean_content.py --corpus product-knowledge
```
### 3. Define the knowledge model
```powershell
python .\src\03_define_knowledge_model.py `
--corpus product-knowledge `
--model-file .\src\templates\knowledge_model.json
```
### 4. Extract entities and relationships
```powershell
python .\src\04_extract_knowledge.py --corpus product-knowledge
```
LLM calls occur outside long database transactions. Each validated chunk result is
committed independently.
### 5. Reconcile knowledge
```powershell
python .\src\05_reconcile_knowledge.py --corpus product-knowledge
```
### 6. Build graph nodes and edges
```powershell
python .\src\06_build_knowledge_graph.py --corpus product-knowledge
```
### 7. Build communities
```powershell
python .\src\07_build_communities.py --corpus product-knowledge
```
### 8. Build the PgVector index
```powershell
python .\src\08_build_retrieval_index.py --corpus product-knowledge
```
This embeds chunks, nodes, edges, and community summaries, validates dimensions,
stores vectors, and creates a dimension-specific HNSW cosine index.
### 9. Retrieve GraphRAG context
```powershell
python .\src\09_query_graphrag.py `
--corpus product-knowledge `
--question "Who created Atlas?" `
--top-k 8
```
The result contains vector `context`, expanded `graph_context`, and traceable
`sources`.
### 10. Generate a cited answer
```powershell
python .\src\10_generate_answer.py `
--corpus product-knowledge `
--question "Who created Atlas?"
```
## Run a durable build
Run stages 1 through 8 under one durable pipeline record:
```powershell
knowledgegraph run `
--corpus product-knowledge `
--from-stage 1 `
--through-stage 8 `
--input-jsonl .\artifacts\mcp-documents.jsonl `
--model-file .\src\templates\knowledge_model.json `
--requested-by local-developer
```
Use `--from-stage` to rebuild only downstream state. The `--reset` option deletes
the selected stage and downstream derived corpus data before execution.
## BFF authorization contract
The existing BFF authenticates callers and evaluates the `KnowledgeGraph` feature.
Independent permission bits are:
```text
Read = 1
Add = 2
Edit = 4
Delete = 8
Execute = 16
```
| Operation | Required mask |
|---|---:|
| Read job/source/retrieval information | `Read` (`1`) |
| Submit a stage-1 source job | `Add | Execute` (`18`) |
| Run downstream build stages | `Execute` (`16`) |
| Query or answer | `Read | Execute` (`17`) |
| Retry a failed job | `Execute` (`16`) |
The BFF forwards a signed internal JWT. Required claims:
```json
{
"iss": "knowledgegraph-bff",
"aud": "knowledgegraph-service",
"sub": "internal-caller-id",
"iat": 1791930000,
"exp": 1791930600,
"jti": "correlation-or-request-id",
"features": {
"KnowledgeGraph": 31
}
}
```
Dev/test uses HS256 and `INTERNAL_JWT_SECRET`. Production deployments can replace
this verifier with the BFF's workload identity, mTLS, or asymmetric/JWKS contract.
Never trust unsigned permission headers.
## Internal API
| Endpoint | Purpose |
|---|---|
| `GET /internal/knowledge-graph/health` | Process liveness |
| `GET /internal/knowledge-graph/ready` | Database and PgVector readiness |
| `POST /internal/knowledge-graph/jobs` | Queue stages 1–8 |
| `GET /internal/knowledge-graph/jobs/{jobId}` | Read durable job/stage status |
| `POST /internal/knowledge-graph/jobs/{jobId}/retry` | Requeue a failed job |
| `POST /internal/knowledge-graph/query` | Synchronous bounded retrieval |
| `POST /internal/knowledge-graph/answer` | Synchronous cited answer |
Example job request:
```json
{
"corpus": "product-knowledge",
"from_stage": 1,
"through_stage": 8,
"input_jsonl": "/app/artifacts/mcp-documents.jsonl",
"model_file": "/app/src/templates/knowledge_model.json",
"reset": false
}
```
Compose bind-mounts local [`artifacts`](artifacts) at `/app/artifacts`, allowing the
worker to read JSONL handoffs placed there. In deployed environments, prefer the
PostgreSQL source contract or a shared managed file volume.
Long-running stages return `202 Accepted` with a job ID. Query and answer requests
remain synchronous and bounded.
## Retry and logging behavior
Model-provider calls retry:
- HTTP 429
- HTTP 500, 502, 503, and 504
- connection failures
- timeouts
Retries honor numeric `Retry-After`, otherwise use bounded exponential backoff with
jitter. Authentication and validation errors are not retried.
Logs are emitted as one JSON object per line and include event names, run IDs,
corpus/stage context where available, attempts, status, and error details. Secrets
are not logged.
## Tests
Portable tests:
```powershell
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
```
Real PostgreSQL/PgVector migration and retrieval test:
```powershell
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
pytest .\tests\integration\test_pgvector_postgres.py
Remove-Item Env:TEST_POSTGRES_URL
```
The suite covers:
- Versioned source validation
- Nested metadata and quoted/comma-containing content
- Configuration parsing
- Chunk validation
- Cross-chunk relationship reconciliation
- Provider retry policy
- Full ten-stage behavior
- Signed BFF authorization and permission failures
- Durable API job and worker execution
- Alembic migration against PostgreSQL
- PgVector extension and HNSW index creation
- Server-side vector retrieval, graph expansion, and citations
Opt-in vector load test:
```powershell
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
$env:RUN_PERFORMANCE_TESTS = "1"
$env:PERF_VECTOR_ITEM_COUNT = "10000"
$env:PERF_MAX_QUERY_SECONDS = "2.0"
pytest .\tests\performance\test_vector_query_load.py
```
Set the item count and latency threshold to the capacity target for the dev/test
environment. Performance checks are excluded by default because machine resources
vary.
CI is defined in [`.github/workflows/ci.yml`](.github/workflows/ci.yml) and compiles,
tests, validates Compose, exercises real PgVector, and builds the application image.
## Persistent format policy
- PostgreSQL is the canonical operational store.
- JSON Lines is the preferred file handoff/checkpoint format.
- CSV is allowed only for explicitly defined flat import/export views.
CSV safely supports commas, quotes, and line breaks when written by a compliant CSV
library. It remains unsuitable as the canonical graph format because aliases,
metadata, source ID arrays, embeddings, and graph properties are nested and typed.
Never construct CSV by joining strings with commas.
## RAG database alternatives
### PostgreSQL with PgVector
Implemented default. Choose it for SQL transactions, colocated provenance and
vectors, self-managed deployment, and tunable HNSW retrieval. Graph traversal uses
indexed relational edges rather than a native property-graph language.
Official reference: [PgVector](https://github.com/pgvector/pgvector).
### Azure Cosmos DB for NoSQL
Choose it for managed/serverless document storage, vector indexing, and first-party
vector plus BM25 hybrid search. Cosmos DB's Gremlin API is separate from the NoSQL
vector feature; use document adjacency or a separate graph store.
References:
- [Cosmos DB vector search](https://learn.microsoft.com/azure/cosmos-db/vector-search)
- [Cosmos DB hybrid search](https://learn.microsoft.com/azure/cosmos-db/gen-ai/hybrid-search)
- [Cosmos DB Gremlin API](https://learn.microsoft.com/azure/cosmos-db/gremlin/overview)
### Neo4j
Choose it when graph topology, paths, and multi-hop traversal are primary retrieval
signals. Neo4j combines native property graphs, Cypher, vector indexes, and
graph-aware hybrid retrieval. It requires a dedicated backend adapter; changing
`DATABASE_URL` is not sufficient. Confirm version and edition requirements for the
selected vector and unified-search features.
References:
- [Neo4j vector indexes](https://neo4j.com/docs/cypher-manual/current/indexes/search-indexes/vector-indexes/)
- [Neo4j hybrid search](https://neo4j.com/developer/genai-ecosystem/hybrid-search/)
Other credible options:
- [Amazon Neptune Analytics](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html)
for AWS-managed graph/vector analytics
- [Azure AI Search](https://learn.microsoft.com/azure/search/hybrid-search-overview)
as a secondary managed hybrid index without a native graph model
## Operational guidance
- Keep the API and worker on internal networks accessible only to the BFF and
operators.
- Replace all dev secrets before shared deployment.
- Use a separate PostgreSQL role for migrations and a least-privilege runtime role.
- Back up PostgreSQL before destructive resets or migrations.
- Monitor queued/running/failed run counts, stage duration, LLM retries, database
pool usage, vector latency, and retrieval result counts.
- Set `WORKER_STALE_RUN_SECONDS` above the longest expected single stage.
- Rebuild stage 8 after changing embedding model or dimensions.
- Maintain a golden question/citation set to detect retrieval-quality regressions.
## Troubleshooting
### Database migrations are required
```powershell
alembic current
alembic upgrade head
```
### `LLM_API_KEY is required`
Set `LLM_API_KEY` in `.env` and restart API/worker containers.
### Embedding dimension mismatch
Set `LLM_EMBEDDING_DIMENSIONS` to the exact provider output dimension, then reset
and rerun stage 8.
### Source handoff rejected
Validate every JSONL line against schema version `1.0`. Confirm `corpus`,
`external_id`, URL, body, and `retrieved_at`.
### Job remains queued
```powershell
podman compose ps worker
podman compose logs --tail 100 worker
```
### PostgreSQL/PgVector is not ready
```powershell
podman compose ps postgres
podman compose logs --tail 100 postgres migrate
```
### Port conflict
Change `POSTGRES_PORT` or `API_PORT` in `.env`, then recreate the Compose stack.