2.7 KiB
Operations and troubleshooting
Health and readiness
/internal/knowledge-graph/healthchecks process liveness./internal/knowledge-graph/readychecks database and PgVector readiness.podman compose psshows service and health state.podman compose logs --follow api workerstreams structured JSON logs.
Logs include event names, run IDs, corpus/stage context, attempts, status, and error details. Secrets are not intended to be logged.
Common failures
Migration does not complete
Check PostgreSQL health, credentials, and DATABASE_URL. Confirm the PgVector
image is used and inspect:
podman compose logs migrate postgres
alembic current
Do not enable automatic schema creation in a shared environment.
Source validation fails
Check schema version, required fields, ISO timestamp offset, unique
external_id, and corpus. The error identifies the source file and line.
LLM requests fail
Verify LLM_BASE_URL, key, model names, and embedding dimensions. 429 and
transient server/network errors retry; authentication and validation errors fail
immediately. Check provider quota and Retry-After behavior before increasing
retry settings.
Vector index or retrieval fails
Confirm the vector extension exists, the provider returns exactly
LLM_EMBEDDING_DIMENSIONS values, and stage 8 completed for the selected
corpus. SQLite cannot validate PgVector behavior.
Job remains queued
Check worker logs and the corpus advisory lock. Another build may be active.
The worker requeues stale runs after WORKER_STALE_RUN_SECONDS; set this above
the maximum expected stage duration.
Job fails after partial progress
Read the job and stage error, repair the source/configuration, then retry. Retry resumes after the last successful stage. Use a reset rebuild if partial side effects or changed model settings make the checkpoint unsafe.
Known limitations
- Only source schema
1.0is accepted. ANALYTICS_DATABASE_URLis configured but unused by the current pipeline.- Source table names are simple unqualified names.
- Stage-level recovery is not an all-stage transaction.
- Name normalization can raise on conflicting entity types.
- LLM shape validation does not establish factual correctness.
- Answer citations are prompt-enforced, not independently checked.
- No observed API rate limiting or pagination middleware.
- The migration set currently contains one initial revision.
Operational safety
- Use read-only credentials for the external source database.
- Keep API and worker on private network paths behind the BFF.
- Rotate all development secrets before deployment.
- Back up PostgreSQL before destructive resets or migration changes.
- Treat
podman compose down --volumesas destructive.