From 82a5a0e55746d9f8212a7de928467fdcae21bf89 Mon Sep 17 00:00:00 2001 From: GAJ3T Date: Thu, 15 Oct 2026 10:20:38 -0400 Subject: [PATCH] docs updated --- README.md | 3 + docs/README.md | 45 ++++++++++ docs/api.md | 71 ++++++++++++++++ docs/architecture.md | 88 ++++++++++++++++++++ docs/configuration.md | 79 ++++++++++++++++++ docs/data-model.md | 38 +++++++++ docs/diagrams/c4-code.drawio | 1 + docs/diagrams/c4-component.drawio | 1 + docs/diagrams/c4-container.drawio | 1 + docs/diagrams/c4-context.drawio | 1 + docs/module-reference.md | 46 +++++++++++ docs/operations.md | 76 +++++++++++++++++ docs/overview.md | 68 ++++++++++++++++ docs/runbook.md | 131 ++++++++++++++++++++++++++++++ docs/source-contract.md | 72 ++++++++++++++++ docs/testing.md | 47 +++++++++++ 16 files changed, 768 insertions(+) create mode 100644 docs/README.md create mode 100644 docs/api.md create mode 100644 docs/architecture.md create mode 100644 docs/configuration.md create mode 100644 docs/data-model.md create mode 100644 docs/diagrams/c4-code.drawio create mode 100644 docs/diagrams/c4-component.drawio create mode 100644 docs/diagrams/c4-container.drawio create mode 100644 docs/diagrams/c4-context.drawio create mode 100644 docs/module-reference.md create mode 100644 docs/operations.md create mode 100644 docs/overview.md create mode 100644 docs/runbook.md create mode 100644 docs/source-contract.md create mode 100644 docs/testing.md diff --git a/README.md b/README.md index 574e893..539820b 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,9 @@ The API is internal only and is not intended for direct customer access. | [`Containerfile`](Containerfile) | Non-root application image | | [`.env.example`](.env.example) | Runtime configuration template | +Detailed architecture, API, runbook, configuration, data model, testing, and +editable C4 diagrams are available in [`docs/`](docs/README.md). + ## Prerequisites - Python 3.11 or newer diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..831a7ac --- /dev/null +++ b/docs/README.md @@ -0,0 +1,45 @@ +# KnowledgeGraph GraphRAG documentation + +This folder is the operational and technical reference for the KnowledgeGraph +GraphRAG proof of concept. The repository README remains the shortest getting +started path; these documents explain the contracts behind it. + +## Read this first + +| Document | Use it for | +|---|---| +| [System overview](./overview.md) | Scope, boundaries, stage flow, and key design decisions | +| [Architecture](./architecture.md) | Runtime topology, stage contracts, persistence, and retrieval | +| [Runbook](./runbook.md) | Step-by-step local setup, ingestion, rebuilds, recovery, and shutdown | +| [Configuration](./configuration.md) | Environment variables, defaults, and production guidance | +| [Source contract](./source-contract.md) | JSONL and PostgreSQL handoff schemas | +| [API reference](./api.md) | Internal endpoints, JWT claims, permissions, and examples | +| [Data model](./data-model.md) | Checkpoint tables and stage ownership | +| [Testing](./testing.md) | Test commands, prerequisites, and performance checks | +| [Operations and troubleshooting](./operations.md) | Logs, health checks, failures, retries, and known limitations | +| [Module reference](./module-reference.md) | Python modules, stage entry points, and responsibilities | + +## C4 diagrams + +The diagrams are editable in [diagrams.net/draw.io](https://app.diagrams.net/). + +| C4 level | Diagram | +|---|---| +| Context | [c4-context.drawio](./diagrams/c4-context.drawio) | +| Container | [c4-container.drawio](./diagrams/c4-container.drawio) | +| Component | [c4-component.drawio](./diagrams/c4-component.drawio) | +| Code | [c4-code.drawio](./diagrams/c4-code.drawio) | + +## Documentation scope + +The documentation is based on the implementation, tests, migration, Compose +definition, CI workflow, templates, and repository README. It documents current +behavior rather than promising features that are not implemented. In particular: + +- source collection is external to this repository; +- stages 1-8 are durable worker jobs; +- query and answer are synchronous API/stage operations; +- PostgreSQL with PgVector is the deployment target; +- SQLite is supported for isolated tests, not production retrieval; +- LLM output is schema-validated but not independently fact-checked. + diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..b143544 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,71 @@ +# Internal API reference + +All routes are under `/internal/knowledge-graph`. Health is unauthenticated; +other routes require an internal signed JWT from the BFF. + +## Endpoints + +| Method and path | Permission | Behavior | +|---|---:|---| +| `GET /health` | none | process liveness | +| `GET /ready` | none | database and PgVector readiness | +| `POST /jobs` | stage 1: `18`; downstream: `16` | queue stages 1-8 | +| `GET /jobs/{jobId}` | `1` | durable job and stage status | +| `POST /jobs/{jobId}/retry` | `16` | requeue a failed job | +| `POST /query` | `17` | synchronous retrieval | +| `POST /answer` | `17` | synchronous retrieval plus cited answer | + +Permission bits are `Read=1`, `Add=2`, `Edit=4`, `Delete=8`, and `Execute=16`. + +## Job request + +```json +{ + "corpus": "product-knowledge", + "from_stage": 1, + "through_stage": 8, + "input_jsonl": "/app/artifacts/mcp-documents.jsonl", + "model_file": "/app/src/templates/knowledge_model.json", + "reset": false +} +``` + +Long-running job submission returns `202 Accepted` with a `job_id`. Query and +answer remain synchronous and bounded. + +## Query and answer requests + +```json +{ + "corpus": "product-knowledge", + "question": "Who created Atlas?", + "top_k": 8 +} +``` + +Retrieval returns vector `context`, bounded `graph_context`, and traceable +`sources`. Answer generation asks the LLM to cite those sources as `[S1]`, +`[S2]`, and so on. + +## JWT contract + +The verifier expects HS256 in the current implementation and validates `iss`, +`aud`, `sub`, `iat`, `exp`, `jti`, and a feature permission mask: + +```json +{ + "iss": "knowledgegraph-bff", + "aud": "knowledgegraph-service", + "sub": "internal-caller-id", + "iat": 1791930000, + "exp": 1791930600, + "jti": "request-id", + "features": { + "KnowledgeGraph": 31 + } +} +``` + +The BFF remains the policy enforcement point. Do not substitute unsigned +headers for the token. + diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..e220c6c --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,88 @@ +# Architecture + +## Logical architecture + +```text +Web-Access/Data-Store MCP + | + | JSONL file or PostgreSQL source table + v +StageContext + numbered stages + | + +--> PostgreSQL checkpoint tables + +--> PgVector retrieval_items + HNSW index + +--> structured JSON logs and artifacts + | + +--> OpenAI-compatible chat/embedding provider + +BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages + | + +--> query/answer stages +``` + +See the editable [C4 context diagram](./diagrams/c4-context.drawio) and +[C4 container diagram](./diagrams/c4-container.drawio) for the system boundary. + +## Stage contracts + +| Stage | Reads | Writes | Result | +|---:|---|---|---| +| 1 | JSONL or source table | `raw_documents`, `documents` | normalized documents | +| 2 | documents | `chunks` | cleaned, deduplicated chunks | +| 3 | settings/model file | `knowledge_models` | entity and relationship vocabulary | +| 4 | chunks, model | `extractions` | validated LLM extraction payloads | +| 5 | extractions | `resolved_entities`, `resolved_relationships` | canonical entities and links | +| 6 | resolved data | `nodes`, `edges` | graph materialization | +| 7 | graph | `communities` | detected communities and LLM summaries | +| 8 | chunks, graph, communities | `search_items` | embeddings and HNSW index | +| 9 | search index, graph, documents | none | vector context, graph context, sources | +| 10 | stage 9 result | none | cited answer text and sources | + +Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds +the question, retrieves `top_k` items, expands graph neighbors for the +configured number of hops and limit, and resolves source URLs. + +## Durable execution + +`pipeline_runs` and `stage_runs` are separate from stage data. A run can be: + +```text +queued -> running -> succeeded + \-> failed -> queued (retry) +``` + +The worker: + +1. initializes and verifies the schema; +2. requeues stale runs older than `WORKER_STALE_RUN_SECONDS`; +3. claims one queued run using database locking; +4. takes a per-corpus PostgreSQL advisory lock; +5. executes the selected stage range; +6. persists stage status and outputs; +7. releases the lock. + +If a corpus lock is unavailable, execution is deferred and the job remains +queued. Retry resumes after the last successful stage unless the caller uses +`--reset` or otherwise clears downstream state. + +## Deployment topology + +`compose.yaml` starts PostgreSQL, waits for its health check, runs the one-shot +`migrate` service, then starts `api` and `worker`. The API and worker share the +application environment and bind-mount `./artifacts` into `/app/artifacts`. + +The [C4 container diagram](./diagrams/c4-container.drawio) shows these +containers and external dependencies. The [component diagram](./diagrams/c4-component.drawio) +maps the application package, and the [code diagram](./diagrams/c4-code.drawio) +maps the numbered stage pipeline. + +## Failure and consistency model + +- Source records are rejected before they become normalized documents if their + schema, required fields, or corpus are invalid. +- A completed stage is a checkpoint, not a transaction spanning the whole build. +- A failed stage may leave partial stage side effects; reset the affected stage + and downstream state when a clean rebuild is required. +- PostgreSQL is required to exercise PgVector and HNSW behavior. +- Embedding vector length must equal `LLM_EMBEDDING_DIMENSIONS`. + diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..4d3a9a2 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,79 @@ +# Configuration reference + +`Settings` reads environment variables and `.env` values. The complete starter +set is in [`.env.example`](../.env.example). Values below are grouped by +responsibility; names are case-insensitive through Pydantic settings. + +## Database and source + +| Variable | Purpose | Default | +|---|---|---| +| `DATABASE_URL` | canonical checkpoint database | PostgreSQL on localhost | +| `SOURCE_DATABASE_URL` | optional external source database | empty | +| `SOURCE_TABLE_NAME` | source table | `external_documents` | +| `ANALYTICS_DATABASE_URL` | reserved analytics setting | empty | +| `AUTO_CREATE_SCHEMA` | test-only schema creation | `false` | +| `ARTIFACT_DIR` | local handoff/artifact directory | `artifacts` | + +## Corpus and model + +| Variable | Purpose | Default | +|---|---|---| +| `CORPUS_NAME` | corpus namespace | `default-corpus` | +| `ENTITY_TYPES` | comma-separated allowed entity types | Person, Organization, Product, Concept, Event, Location | +| `RELATIONSHIP_TYPES` | comma-separated allowed edge types | AFFILIATED_WITH, CREATED, DEPENDS_ON, MENTIONS, PART_OF, RELATED_TO | +| `NODE_TEMPLATE` | JSON extraction shape | template in `.env.example` | +| `EDGE_TEMPLATE` | JSON extraction shape | template in `.env.example` | +| `EXTRACTION_INSTRUCTIONS` | additional extraction guardrail | source-grounded extraction | + +Use [the model template](../src/templates/knowledge_model.json) when invoking +stage 3 with `--model-file`. + +## LLM provider + +| Variable | Purpose | Default | +|---|---|---| +| `LLM_BASE_URL` | OpenAI-compatible API root | `https://api.openai.com/v1` | +| `LLM_API_KEY` | provider credential | empty | +| `LLM_CHAT_MODEL` | chat/completion model | `gpt-4.1-mini` | +| `LLM_EMBEDDING_MODEL` | embedding model | `text-embedding-3-small` | +| `LLM_EMBEDDING_DIMENSIONS` | expected vector length | `1536` | +| `LLM_TIMEOUT_SECONDS` | request timeout | `60` | +| `LLM_MAX_RETRIES` | retry count | `4` | +| `LLM_RETRY_INITIAL_SECONDS` | initial backoff | `1` | +| `LLM_RETRY_MAX_SECONDS` | maximum backoff | `30` | + +Retries cover 429, 500/502/503/504, connection failures, and timeouts. +Numeric `Retry-After` is honored. Authentication and validation failures are +not retried. + +## Processing and retrieval + +| Variable | Purpose | Default | +|---|---|---:| +| `CHUNK_SIZE` | chunk target size | `1200` | +| `CHUNK_OVERLAP` | chunk overlap | `150` | +| `COMMUNITY_RESOLUTION` | NetworkX community resolution | `1.0` | +| `RETRIEVAL_TOP_K` | vector result count | `8` | +| `RETRIEVAL_GRAPH_HOPS` | graph expansion depth | `2` | +| `RETRIEVAL_GRAPH_LIMIT` | graph expansion cap | `50` | +| `RETRIEVAL_HNSW_EF_SEARCH` | PgVector search breadth | `100` | + +`CHUNK_OVERLAP` must be smaller than `CHUNK_SIZE`. Embedding dimensions must +match the provider output and the PgVector index dimension. + +## Security and worker + +| Variable | Purpose | +|---|---| +| `INTERNAL_JWT_SECRET` | HS256 verification secret | +| `INTERNAL_JWT_ISSUER` | expected JWT issuer | +| `INTERNAL_JWT_AUDIENCE` | expected JWT audience | +| `INTERNAL_FEATURE_NAME` | feature key containing permission bits | +| `LOG_LEVEL` | structured log level | +| `WORKER_POLL_SECONDS` | idle worker polling interval | +| `WORKER_STALE_RUN_SECONDS` | stale-run recovery threshold | + +Change every development secret before shared or production use. Prefer workload +identity, mTLS, or asymmetric/JWKS verification in production. + diff --git a/docs/data-model.md b/docs/data-model.md new file mode 100644 index 0000000..682fe92 --- /dev/null +++ b/docs/data-model.md @@ -0,0 +1,38 @@ +# Data model and checkpoint ownership + +The initial Alembic revision creates the SQLAlchemy metadata and the PgVector +extension. The models live in [`src/graphrag_platform/models.py`](../src/graphrag_platform/models.py). + +## Pipeline data + +| Table/model | Owner stage | Meaning | +|---|---:|---| +| `raw_documents` | 1 | original source payload and provenance | +| `documents` | 1 | normalized text and source identity | +| `chunks` | 2 | cleaned, overlapping retrieval units | +| `knowledge_models` | 3 | configured entity/relationship vocabulary and templates | +| `extractions` | 4 | validated LLM payload per chunk | +| `resolved_entities` | 5 | canonical entities and aliases | +| `resolved_relationships` | 5 | relationships linked to canonical keys | +| `nodes` | 6 | graph nodes | +| `edges` | 6 | graph edges | +| `communities` | 7 | detected groups and summaries | +| `search_items` | 8 | text, metadata, source URL, and embedding | + +## Run tracking + +| Table/model | Purpose | +|---|---| +| `pipeline_runs` | requested range, status, attempt, caller, correlation, and errors | +| `stage_runs` | per-stage status, input/output, timestamps, and error details | + +Stage data is corpus-scoped. Clear or rebuild from the first invalid stage +through downstream derived data; do not manually delete unrelated corpora. + +## Indexing + +PostgreSQL creates a dimension-specific HNSW cosine index on retrieval items. +SQLite stores vectors as JSON and uses in-process cosine similarity. The latter +is useful for tests but does not validate production query plans or PgVector +behavior. + diff --git a/docs/diagrams/c4-code.drawio b/docs/diagrams/c4-code.drawio new file mode 100644 index 0000000..93d9129 --- /dev/null +++ b/docs/diagrams/c4-code.drawio @@ -0,0 +1 @@ + diff --git a/docs/diagrams/c4-component.drawio b/docs/diagrams/c4-component.drawio new file mode 100644 index 0000000..35be60b --- /dev/null +++ b/docs/diagrams/c4-component.drawio @@ -0,0 +1 @@ + diff --git a/docs/diagrams/c4-container.drawio b/docs/diagrams/c4-container.drawio new file mode 100644 index 0000000..bdeadab --- /dev/null +++ b/docs/diagrams/c4-container.drawio @@ -0,0 +1 @@ + diff --git a/docs/diagrams/c4-context.drawio b/docs/diagrams/c4-context.drawio new file mode 100644 index 0000000..30e168c --- /dev/null +++ b/docs/diagrams/c4-context.drawio @@ -0,0 +1 @@ + diff --git a/docs/module-reference.md b/docs/module-reference.md new file mode 100644 index 0000000..7336e68 --- /dev/null +++ b/docs/module-reference.md @@ -0,0 +1,46 @@ +# Module reference + +## Platform package + +| Module | Responsibility | +|---|---| +| `config.py` | typed settings, environment parsing, model definition | +| `models.py` | SQLAlchemy entities and checkpoint schema | +| `storage.py` | engine/session, schema checks, persistence, vector index | +| `sources.py` | strict JSONL and PostgreSQL source readers | +| `llm.py` | OpenAI-compatible completion and embedding client | +| `runtime.py` | `StageContext` construction and initialization | +| `stage_loader.py` | numbered stage module lookup | +| `pipeline.py` | compatibility facade for stage methods | +| `orchestrator.py` | durable runs, claiming, resume, stale recovery, locks | +| `auth.py` | JWT verification and permission masks | +| `api.py` | FastAPI app, request models, routes, authorization | +| `worker.py` | polling worker and one-run execution | +| `cli.py` | stage and durable-run command line entry points | +| `observability.py` | structured JSON logging | + +## Numbered stages + +| Module | Main responsibility | +|---|---| +| `01_normalize_content.py` | validate handoff, extract text, persist raw/normalized documents | +| `02_clean_content.py` | clean, deduplicate, and chunk | +| `03_define_knowledge_model.py` | persist entity/relationship vocabulary and templates | +| `04_extract_knowledge.py` | call LLM, validate extraction, persist per-chunk results | +| `05_reconcile_knowledge.py` | canonicalize names and resolve cross-chunk relationships | +| `06_build_knowledge_graph.py` | materialize nodes and edges | +| `07_build_communities.py` | detect communities and summarize them | +| `08_build_retrieval_index.py` | embed chunks/graph/community items and build HNSW | +| `09_query_graphrag.py` | vector retrieval, graph expansion, source resolution | +| `10_generate_answer.py` | generate answer with source citations | + +Each numbered module exposes a `run` function and command-line adapter. Stages +1-5 also expose focused validation/transformation helpers; stage 9 exposes +retrieval helpers such as cosine ranking and graph expansion. + +## Entry points + +- `knowledgegraph` -> `graphrag_platform.cli:main_all` +- `knowledgegraph-worker` -> `graphrag_platform.worker:main` +- `uvicorn graphrag_platform.api:app` + diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..d1316d9 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,76 @@ +# Operations and troubleshooting + +## Health and readiness + +- `/internal/knowledge-graph/health` checks process liveness. +- `/internal/knowledge-graph/ready` checks database and PgVector readiness. +- `podman compose ps` shows service and health state. +- `podman compose logs --follow api worker` streams structured JSON logs. + +Logs include event names, run IDs, corpus/stage context, attempts, status, and +error details. Secrets are not intended to be logged. + +## Common failures + +### Migration does not complete + +Check PostgreSQL health, credentials, and `DATABASE_URL`. Confirm the PgVector +image is used and inspect: + +```powershell +podman compose logs migrate postgres +alembic current +``` + +Do not enable automatic schema creation in a shared environment. + +### Source validation fails + +Check schema version, required fields, ISO timestamp offset, unique +`external_id`, and corpus. The error identifies the source file and line. + +### LLM requests fail + +Verify `LLM_BASE_URL`, key, model names, and embedding dimensions. 429 and +transient server/network errors retry; authentication and validation errors fail +immediately. Check provider quota and `Retry-After` behavior before increasing +retry settings. + +### Vector index or retrieval fails + +Confirm the `vector` extension exists, the provider returns exactly +`LLM_EMBEDDING_DIMENSIONS` values, and stage 8 completed for the selected +corpus. SQLite cannot validate PgVector behavior. + +### Job remains queued + +Check worker logs and the corpus advisory lock. Another build may be active. +The worker requeues stale runs after `WORKER_STALE_RUN_SECONDS`; set this above +the maximum expected stage duration. + +### Job fails after partial progress + +Read the job and stage error, repair the source/configuration, then retry. Retry +resumes after the last successful stage. Use a reset rebuild if partial side +effects or changed model settings make the checkpoint unsafe. + +## Known limitations + +- Only source schema `1.0` is accepted. +- `ANALYTICS_DATABASE_URL` is configured but unused by the current pipeline. +- Source table names are simple unqualified names. +- Stage-level recovery is not an all-stage transaction. +- Name normalization can raise on conflicting entity types. +- LLM shape validation does not establish factual correctness. +- Answer citations are prompt-enforced, not independently checked. +- No observed API rate limiting or pagination middleware. +- The migration set currently contains one initial revision. + +## Operational safety + +- Use read-only credentials for the external source database. +- Keep API and worker on private network paths behind the BFF. +- Rotate all development secrets before deployment. +- Back up PostgreSQL before destructive resets or migration changes. +- Treat `podman compose down --volumes` as destructive. + diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..de674ec --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,68 @@ +# System overview + +## Purpose + +KnowledgeGraph GraphRAG turns externally collected documents into a +source-grounded knowledge graph and vector retrieval index. It then combines +semantic retrieval with bounded graph expansion to produce context and cited +answers. + +The collection boundary is deliberate: Web-Access MCP and Data-Store MCP +discover and persist source material. This repository starts at their versioned +handoff. + +## End-to-end flow + +```text +external source handoff + -> 1 normalize + -> 2 clean, deduplicate, chunk + -> 3 persist knowledge model + -> 4 extract entities and relationships + -> 5 reconcile aliases and duplicates + -> 6 materialize nodes and edges + -> 7 detect and summarize communities + -> 8 embed and index + -> 9 retrieve vector + graph + source context + -> 10 generate cited answer +``` + +Stages 1-8 persist durable checkpoints in PostgreSQL. A later stage can be +re-run without repeating valid upstream work. A durable run records stage +selection, attempts, configuration fingerprint, status, timestamps, and errors. + +## Runtime boundary + +The deployment contains: + +1. PostgreSQL 16 with the PgVector extension. +2. An Alembic migration job. +3. An internal FastAPI service. +4. A PostgreSQL-backed worker. +5. An existing BFF that authenticates callers and enforces feature permissions. +6. An OpenAI-compatible chat and embedding provider. + +The API is an internal service and is not intended for direct customer access. + +## Important design decisions + +- PostgreSQL is the canonical checkpoint store. +- PgVector performs server-side cosine search using a dimension-specific HNSW + index. +- SQLite is a local-test fallback with JSON embeddings and in-process search. +- LLM calls happen outside long database transactions; validated chunk results + are committed independently. +- A PostgreSQL advisory lock permits one active build per corpus. +- The worker can recover stale runs and resume after the last successful stage. +- The BFF supplies a short-lived signed internal JWT; unsigned permission + headers are not trusted. + +## Non-goals and boundaries + +- Crawling and source discovery are not implemented here. +- The API does not provide customer-facing authentication or authorization. +- Query and answer are not queued worker jobs. +- Citation correctness is requested through the answer prompt, not independently + verified. +- Entity reconciliation is normalized-name based, not semantic entity matching. + diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 0000000..c7e025c --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,131 @@ +# Step-by-step runbook + +Commands use PowerShell on Windows and assume the repository root is current. + +## 1. Prepare tools and environment + +Prerequisites: Python 3.11+, Podman 5+, a running `podman machine`, and an +OpenAI-compatible chat/embedding endpoint. + +```powershell +Copy-Item .env.example .env +py -3.12 -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -e ".[test]" +``` + +Edit `.env`: set `POSTGRES_PASSWORD`, matching `DATABASE_URL` credentials, +`LLM_API_KEY`, model names/dimensions, and a random +`INTERNAL_JWT_SECRET` of at least 32 bytes. + +## 2. Start the stack + +```powershell +podman machine start +podman compose up --detach --build +podman compose ps +Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health +Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready +``` + +Compose starts PostgreSQL, applies migrations, and then starts API and worker. +The migration service must complete before the application services start. + +## 3. Supply a source handoff + +Place a UTF-8 JSONL file in `.\artifacts`, or configure a PostgreSQL source +table. Validate it against the [source contract](./source-contract.md). For a +file handoff, use `.\artifacts\mcp-documents.jsonl`. + +## 4. Run a durable build + +```powershell +knowledgegraph run ` + --corpus product-knowledge ` + --from-stage 1 ` + --through-stage 8 ` + --input-jsonl .\artifacts\mcp-documents.jsonl ` + --model-file .\src\templates\knowledge_model.json ` + --requested-by local-developer +``` + +Watch progress: + +```powershell +podman compose logs --follow api worker +``` + +## 5. Query and answer + +For direct local stage execution: + +```powershell +python .\src\09_query_graphrag.py ` + --corpus product-knowledge ` + --question "Who created Atlas?" ` + --top-k 8 + +python .\src\10_generate_answer.py ` + --corpus product-knowledge ` + --question "Who created Atlas?" +``` + +For the API, obtain a BFF-issued JWT and call `/query` or `/answer` as described +in the [API reference](./api.md). + +## 6. Rebuild downstream data + +Use `--from-stage` to avoid repeating valid upstream work. Use `--reset` when +the selected stage or any downstream data is inconsistent: + +```powershell +knowledgegraph run ` + --corpus product-knowledge ` + --from-stage 4 ` + --through-stage 8 ` + --reset ` + --requested-by rebuild +``` + +Do not run two builds for the same corpus concurrently; the PostgreSQL advisory +lock will defer one, but coordinating callers is clearer. + +## 7. Apply or inspect migrations + +```powershell +alembic current +alembic history +alembic upgrade head +``` + +`AUTO_CREATE_SCHEMA=true` is for isolated tests only. Review generated +migrations before applying them. + +## 8. Run tests + +```powershell +pytest --ignore=.\tests\integration\test_pgvector_postgres.py +``` + +For the real PgVector integration: + +```powershell +$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph" +pytest .\tests\integration\test_pgvector_postgres.py +Remove-Item Env:TEST_POSTGRES_URL +``` + +## 9. Stop or reset local services + +```powershell +podman compose down +``` + +This retains the database volume. To delete all local PostgreSQL data: + +```powershell +podman compose down --volumes +``` + +Volume removal is destructive. diff --git a/docs/source-contract.md b/docs/source-contract.md new file mode 100644 index 0000000..1c9c60b --- /dev/null +++ b/docs/source-contract.md @@ -0,0 +1,72 @@ +# External source contract + +## JSON Lines + +Each UTF-8 line must be a JSON object with schema version `1.0`. + +```json +{ + "schema_version": "1.0", + "external_id": "stable-source-record-id", + "corpus": "product-knowledge", + "url": "https://docs.example.com/article", + "body": "
Article text
", + "content_type": "text/html", + "status_code": 200, + "retrieved_at": "2026-10-13T18:00:00Z", + "metadata": { + "collector": "Web-Access", + "stored_by": "Data-Store" + } +} +``` + +Required fields: + +- `schema_version`: exactly `1.0`; +- `external_id`: stable and unique within a corpus; +- `url`; +- `body`; +- `retrieved_at`: ISO 8601 timestamp with an offset. + +Defaulted fields: + +- `content_type`: `text/html`; +- `status_code`: `200`; +- `metadata`: `{}`. + +`corpus`, when provided, must equal the selected runtime corpus. Unknown fields +and unsupported schema versions are rejected. Validation errors include the +file and line number. See the canonical template at +[`src/templates/mcp_documents.jsonl`](../src/templates/mcp_documents.jsonl) and +the acceptance fixture at [`tests/fixtures/external_documents.jsonl`](../tests/fixtures/external_documents.jsonl). + +## PostgreSQL source table + +Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to use a PostgreSQL handoff. +The table must contain: + +```sql +CREATE TABLE external_documents ( + external_id text PRIMARY KEY, + schema_version text NOT NULL DEFAULT '1.0', + corpus text, + url text NOT NULL, + body text NOT NULL, + content_type text NOT NULL DEFAULT 'text/html', + status_code integer NOT NULL DEFAULT 200, + retrieved_at timestamptz NOT NULL, + metadata jsonb NOT NULL DEFAULT '{}'::jsonb +); +``` + +Additional columns are ignored. Give the application account read-only access. +Use either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both. + +## Normalization behavior + +Stage 1 stores raw provenance and a normalized document. HTML is extracted with +Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes +whitespace, detects duplicate content by SHA-256 hash, and creates overlapping +chunks. + diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..c0da51e --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,47 @@ +# Testing and validation + +## Test layers + +| Layer | Coverage | +|---|---| +| Unit | settings, source validation, chunk constraints, reconciliation, provider retry policy | +| Integration | all ten stages, API/worker jobs, repair/retry/resume | +| PostgreSQL integration | migrations, PgVector extension, HNSW index, graph retrieval, advisory lock | +| Performance | configurable vector query load against real PostgreSQL | + +The deterministic `FakeLLM` in `tests/conftest.py` keeps portable tests +repeatable. It is not a quality or latency substitute for a production model. + +## Local validation + +```powershell +pytest --ignore=.\tests\integration\test_pgvector_postgres.py +python -m compileall -q src tests migrations +python -m pip check +``` + +The CI workflow also validates Compose configuration and builds the +`Containerfile` image. + +## PostgreSQL integration + +Set `TEST_POSTGRES_URL` to a PgVector-enabled database. The migration test +resets the schema, creates the external source table, runs stages 1-9, checks +the vector extension and HNSW definition, and verifies advisory-lock behavior. + +## Performance benchmark + +The benchmark is skipped unless both `TEST_POSTGRES_URL` and +`RUN_PERFORMANCE_TESTS=1` are set: + +```powershell +$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph" +$env:RUN_PERFORMANCE_TESTS = "1" +$env:PERF_VECTOR_ITEM_COUNT = "10000" +$env:PERF_MAX_QUERY_SECONDS = "2.0" +pytest .\tests\performance\test_vector_query_load.py -m performance +``` + +The benchmark inserts deterministic three-dimensional vectors and measures +stage 9 retrieval. Results depend on hardware, database configuration, and load. +