RAG-DEMO/docs/operations.md

77 lines
2.7 KiB
Markdown

# Operations and troubleshooting
## Health and readiness
- `/internal/knowledge-graph/health` checks process liveness.
- `/internal/knowledge-graph/ready` checks database and PgVector readiness.
- `podman compose ps` shows service and health state.
- `podman compose logs --follow api worker` streams structured JSON logs.
Logs include event names, run IDs, corpus/stage context, attempts, status, and
error details. Secrets are not intended to be logged.
## Common failures
### Migration does not complete
Check PostgreSQL health, credentials, and `DATABASE_URL`. Confirm the PgVector
image is used and inspect:
```powershell
podman compose logs migrate postgres
alembic current
```
Do not enable automatic schema creation in a shared environment.
### Source validation fails
Check schema version, required fields, ISO timestamp offset, unique
`external_id`, and corpus. The error identifies the source file and line.
### LLM requests fail
Verify `LLM_BASE_URL`, key, model names, and embedding dimensions. 429 and
transient server/network errors retry; authentication and validation errors fail
immediately. Check provider quota and `Retry-After` behavior before increasing
retry settings.
### Vector index or retrieval fails
Confirm the `vector` extension exists, the provider returns exactly
`LLM_EMBEDDING_DIMENSIONS` values, and stage 8 completed for the selected
corpus. SQLite cannot validate PgVector behavior.
### Job remains queued
Check worker logs and the corpus advisory lock. Another build may be active.
The worker requeues stale runs after `WORKER_STALE_RUN_SECONDS`; set this above
the maximum expected stage duration.
### Job fails after partial progress
Read the job and stage error, repair the source/configuration, then retry. Retry
resumes after the last successful stage. Use a reset rebuild if partial side
effects or changed model settings make the checkpoint unsafe.
## Known limitations
- Only source schema `1.0` is accepted.
- `ANALYTICS_DATABASE_URL` is configured but unused by the current pipeline.
- Source table names are simple unqualified names.
- Stage-level recovery is not an all-stage transaction.
- Name normalization can raise on conflicting entity types.
- LLM shape validation does not establish factual correctness.
- Answer citations are prompt-enforced, not independently checked.
- No observed API rate limiting or pagination middleware.
- The migration set currently contains one initial revision.
## Operational safety
- Use read-only credentials for the external source database.
- Keep API and worker on private network paths behind the BFF.
- Rotate all development secrets before deployment.
- Back up PostgreSQL before destructive resets or migration changes.
- Treat `podman compose down --volumes` as destructive.