# Operations and troubleshooting ## Health and readiness - `/internal/knowledge-graph/health` checks process liveness. - `/internal/knowledge-graph/ready` checks database and PgVector readiness. - `podman compose ps` shows service and health state. - `podman compose logs --follow api worker` streams structured JSON logs. Logs include event names, run IDs, corpus/stage context, attempts, status, and error details. Secrets are not intended to be logged. ## Common failures ### Migration does not complete Check PostgreSQL health, credentials, and `DATABASE_URL`. Confirm the PgVector image is used and inspect: ```powershell podman compose logs migrate postgres alembic current ``` Do not enable automatic schema creation in a shared environment. ### Source validation fails Check schema version, required fields, ISO timestamp offset, unique `external_id`, and corpus. The error identifies the source file and line. ### LLM requests fail Verify `LLM_BASE_URL`, key, model names, and embedding dimensions. 429 and transient server/network errors retry; authentication and validation errors fail immediately. Check provider quota and `Retry-After` behavior before increasing retry settings. ### Vector index or retrieval fails Confirm the `vector` extension exists, the provider returns exactly `LLM_EMBEDDING_DIMENSIONS` values, and stage 8 completed for the selected corpus. SQLite cannot validate PgVector behavior. ### Job remains queued Check worker logs and the corpus advisory lock. Another build may be active. The worker requeues stale runs after `WORKER_STALE_RUN_SECONDS`; set this above the maximum expected stage duration. ### Job fails after partial progress Read the job and stage error, repair the source/configuration, then retry. Retry resumes after the last successful stage. Use a reset rebuild if partial side effects or changed model settings make the checkpoint unsafe. ## Known limitations - Only source schema `1.0` is accepted. - `ANALYTICS_DATABASE_URL` is configured but unused by the current pipeline. - Source table names are simple unqualified names. - Stage-level recovery is not an all-stage transaction. - Name normalization can raise on conflicting entity types. - LLM shape validation does not establish factual correctness. - Answer citations are prompt-enforced, not independently checked. - No observed API rate limiting or pagination middleware. - The migration set currently contains one initial revision. ## Operational safety - Use read-only credentials for the external source database. - Keep API and worker on private network paths behind the BFF. - Rotate all development secrets before deployment. - Back up PostgreSQL before destructive resets or migration changes. - Treat `podman compose down --volumes` as destructive.