RAG-DEMO/docs/module-reference.md

47 lines
2.2 KiB
Markdown

# Module reference
## Platform package
| Module | Responsibility |
|---|---|
| `config.py` | typed settings, environment parsing, model definition |
| `models.py` | SQLAlchemy entities and checkpoint schema |
| `storage.py` | engine/session, schema checks, persistence, vector index |
| `sources.py` | strict JSONL and PostgreSQL source readers |
| `llm.py` | OpenAI-compatible completion and embedding client |
| `runtime.py` | `StageContext` construction and initialization |
| `stage_loader.py` | numbered stage module lookup |
| `pipeline.py` | compatibility facade for stage methods |
| `orchestrator.py` | durable runs, claiming, resume, stale recovery, locks |
| `auth.py` | JWT verification and permission masks |
| `api.py` | FastAPI app, request models, routes, authorization |
| `worker.py` | polling worker and one-run execution |
| `cli.py` | stage and durable-run command line entry points |
| `observability.py` | structured JSON logging |
## Numbered stages
| Module | Main responsibility |
|---|---|
| `01_normalize_content.py` | validate handoff, extract text, persist raw/normalized documents |
| `02_clean_content.py` | clean, deduplicate, and chunk |
| `03_define_knowledge_model.py` | persist entity/relationship vocabulary and templates |
| `04_extract_knowledge.py` | call LLM, validate extraction, persist per-chunk results |
| `05_reconcile_knowledge.py` | canonicalize names and resolve cross-chunk relationships |
| `06_build_knowledge_graph.py` | materialize nodes and edges |
| `07_build_communities.py` | detect communities and summarize them |
| `08_build_retrieval_index.py` | embed chunks/graph/community items and build HNSW |
| `09_query_graphrag.py` | vector retrieval, graph expansion, source resolution |
| `10_generate_answer.py` | generate answer with source citations |
Each numbered module exposes a `run` function and command-line adapter. Stages
1-5 also expose focused validation/transformation helpers; stage 9 exposes
retrieval helpers such as cosine ranking and graph expansion.
## Entry points
- `knowledgegraph` -> `graphrag_platform.cli:main_all`
- `knowledgegraph-worker` -> `graphrag_platform.worker:main`
- `uvicorn graphrag_platform.api:app`