47 lines
2.2 KiB
Markdown
47 lines
2.2 KiB
Markdown
# Module reference
|
|
|
|
## Platform package
|
|
|
|
| Module | Responsibility |
|
|
|---|---|
|
|
| `config.py` | typed settings, environment parsing, model definition |
|
|
| `models.py` | SQLAlchemy entities and checkpoint schema |
|
|
| `storage.py` | engine/session, schema checks, persistence, vector index |
|
|
| `sources.py` | strict JSONL and PostgreSQL source readers |
|
|
| `llm.py` | OpenAI-compatible completion and embedding client |
|
|
| `runtime.py` | `StageContext` construction and initialization |
|
|
| `stage_loader.py` | numbered stage module lookup |
|
|
| `pipeline.py` | compatibility facade for stage methods |
|
|
| `orchestrator.py` | durable runs, claiming, resume, stale recovery, locks |
|
|
| `auth.py` | JWT verification and permission masks |
|
|
| `api.py` | FastAPI app, request models, routes, authorization |
|
|
| `worker.py` | polling worker and one-run execution |
|
|
| `cli.py` | stage and durable-run command line entry points |
|
|
| `observability.py` | structured JSON logging |
|
|
|
|
## Numbered stages
|
|
|
|
| Module | Main responsibility |
|
|
|---|---|
|
|
| `01_normalize_content.py` | validate handoff, extract text, persist raw/normalized documents |
|
|
| `02_clean_content.py` | clean, deduplicate, and chunk |
|
|
| `03_define_knowledge_model.py` | persist entity/relationship vocabulary and templates |
|
|
| `04_extract_knowledge.py` | call LLM, validate extraction, persist per-chunk results |
|
|
| `05_reconcile_knowledge.py` | canonicalize names and resolve cross-chunk relationships |
|
|
| `06_build_knowledge_graph.py` | materialize nodes and edges |
|
|
| `07_build_communities.py` | detect communities and summarize them |
|
|
| `08_build_retrieval_index.py` | embed chunks/graph/community items and build HNSW |
|
|
| `09_query_graphrag.py` | vector retrieval, graph expansion, source resolution |
|
|
| `10_generate_answer.py` | generate answer with source citations |
|
|
|
|
Each numbered module exposes a `run` function and command-line adapter. Stages
|
|
1-5 also expose focused validation/transformation helpers; stage 9 exposes
|
|
retrieval helpers such as cosine ranking and graph expansion.
|
|
|
|
## Entry points
|
|
|
|
- `knowledgegraph` -> `graphrag_platform.cli:main_all`
|
|
- `knowledgegraph-worker` -> `graphrag_platform.worker:main`
|
|
- `uvicorn graphrag_platform.api:app`
|
|
|