RAG-POC/docs/operations.md

2.7 KiB

Operations and troubleshooting

Health and readiness

  • /internal/knowledge-graph/health checks process liveness.
  • /internal/knowledge-graph/ready checks database and PgVector readiness.
  • podman compose ps shows service and health state.
  • podman compose logs --follow api worker streams structured JSON logs.

Logs include event names, run IDs, corpus/stage context, attempts, status, and error details. Secrets are not intended to be logged.

Common failures

Migration does not complete

Check PostgreSQL health, credentials, and DATABASE_URL. Confirm the PgVector image is used and inspect:

podman compose logs migrate postgres
alembic current

Do not enable automatic schema creation in a shared environment.

Source validation fails

Check schema version, required fields, ISO timestamp offset, unique external_id, and corpus. The error identifies the source file and line.

LLM requests fail

Verify LLM_BASE_URL, key, model names, and embedding dimensions. 429 and transient server/network errors retry; authentication and validation errors fail immediately. Check provider quota and Retry-After behavior before increasing retry settings.

Vector index or retrieval fails

Confirm the vector extension exists, the provider returns exactly LLM_EMBEDDING_DIMENSIONS values, and stage 8 completed for the selected corpus. SQLite cannot validate PgVector behavior.

Job remains queued

Check worker logs and the corpus advisory lock. Another build may be active. The worker requeues stale runs after WORKER_STALE_RUN_SECONDS; set this above the maximum expected stage duration.

Job fails after partial progress

Read the job and stage error, repair the source/configuration, then retry. Retry resumes after the last successful stage. Use a reset rebuild if partial side effects or changed model settings make the checkpoint unsafe.

Known limitations

  • Only source schema 1.0 is accepted.
  • ANALYTICS_DATABASE_URL is configured but unused by the current pipeline.
  • Source table names are simple unqualified names.
  • Stage-level recovery is not an all-stage transaction.
  • Name normalization can raise on conflicting entity types.
  • LLM shape validation does not establish factual correctness.
  • Answer citations are prompt-enforced, not independently checked.
  • No observed API rate limiting or pagination middleware.
  • The migration set currently contains one initial revision.

Operational safety

  • Use read-only credentials for the external source database.
  • Keep API and worker on private network paths behind the BFF.
  • Rotate all development secrets before deployment.
  • Back up PostgreSQL before destructive resets or migration changes.
  • Treat podman compose down --volumes as destructive.