77 lines
2.7 KiB
Markdown
77 lines
2.7 KiB
Markdown
# Operations and troubleshooting
|
|
|
|
## Health and readiness
|
|
|
|
- `/internal/knowledge-graph/health` checks process liveness.
|
|
- `/internal/knowledge-graph/ready` checks database and PgVector readiness.
|
|
- `podman compose ps` shows service and health state.
|
|
- `podman compose logs --follow api worker` streams structured JSON logs.
|
|
|
|
Logs include event names, run IDs, corpus/stage context, attempts, status, and
|
|
error details. Secrets are not intended to be logged.
|
|
|
|
## Common failures
|
|
|
|
### Migration does not complete
|
|
|
|
Check PostgreSQL health, credentials, and `DATABASE_URL`. Confirm the PgVector
|
|
image is used and inspect:
|
|
|
|
```powershell
|
|
podman compose logs migrate postgres
|
|
alembic current
|
|
```
|
|
|
|
Do not enable automatic schema creation in a shared environment.
|
|
|
|
### Source validation fails
|
|
|
|
Check schema version, required fields, ISO timestamp offset, unique
|
|
`external_id`, and corpus. The error identifies the source file and line.
|
|
|
|
### LLM requests fail
|
|
|
|
Verify `LLM_BASE_URL`, key, model names, and embedding dimensions. 429 and
|
|
transient server/network errors retry; authentication and validation errors fail
|
|
immediately. Check provider quota and `Retry-After` behavior before increasing
|
|
retry settings.
|
|
|
|
### Vector index or retrieval fails
|
|
|
|
Confirm the `vector` extension exists, the provider returns exactly
|
|
`LLM_EMBEDDING_DIMENSIONS` values, and stage 8 completed for the selected
|
|
corpus. SQLite cannot validate PgVector behavior.
|
|
|
|
### Job remains queued
|
|
|
|
Check worker logs and the corpus advisory lock. Another build may be active.
|
|
The worker requeues stale runs after `WORKER_STALE_RUN_SECONDS`; set this above
|
|
the maximum expected stage duration.
|
|
|
|
### Job fails after partial progress
|
|
|
|
Read the job and stage error, repair the source/configuration, then retry. Retry
|
|
resumes after the last successful stage. Use a reset rebuild if partial side
|
|
effects or changed model settings make the checkpoint unsafe.
|
|
|
|
## Known limitations
|
|
|
|
- Only source schema `1.0` is accepted.
|
|
- `ANALYTICS_DATABASE_URL` is configured but unused by the current pipeline.
|
|
- Source table names are simple unqualified names.
|
|
- Stage-level recovery is not an all-stage transaction.
|
|
- Name normalization can raise on conflicting entity types.
|
|
- LLM shape validation does not establish factual correctness.
|
|
- Answer citations are prompt-enforced, not independently checked.
|
|
- No observed API rate limiting or pagination middleware.
|
|
- The migration set currently contains one initial revision.
|
|
|
|
## Operational safety
|
|
|
|
- Use read-only credentials for the external source database.
|
|
- Keep API and worker on private network paths behind the BFF.
|
|
- Rotate all development secrets before deployment.
|
|
- Back up PostgreSQL before destructive resets or migration changes.
|
|
- Treat `podman compose down --volumes` as destructive.
|
|
|