Observability

Purpose

This playbook defines the minimum observability standard for Forward Deployment projects.

Goals

Observability should help the team answer:

• Is the system up?
• Is the client workflow working?
• Are errors increasing?
• Is latency acceptable?
• Are external integrations failing?
• Are AI calls failing, timing out, or producing low-quality results?
• What changed recently?

Minimum Signals

Every production project should have:

• Health endpoint or equivalent uptime check.
• Error tracking for backend failures.
• Deployment history linked from GitHub.
• Logs with enough retention for incident review.
• Alerts for critical failure modes.
• AI-specific metrics if the product depends on AI behavior.

Application Metrics

Track when relevant:

• Request count.
• Error rate.
• Latency.
• Job queue depth.
• Background job failures.
• External API failures.
• Database connection or query pressure.

Infrastructure Metrics

Track when relevant:

• CPU.
• Memory.
• Disk.
• Database storage.
• Database connections.
• Container or process restarts.
• Health check failures.

AI-Specific Metrics

For AI-heavy projects, track:

• Model/provider used.
• Token usage.
• Cost estimate.
• Latency by AI call.
• Tool-call failures.
• Agent fallback rate.
• Human escalation rate.
• Output rejection or correction rate, if available.

Alerting Standard

Alerts should be actionable.

Every alert should answer:

• What is broken?
• Who receives it?
• What is the expected first action?
• What dashboard or log should be checked?

Avoid alerts that are noisy and unactionable. Noisy alerts create alarm blindness.

Release Readiness Requirement

High-risk production releases must identify how the team will detect failure after deployment.

This can be:

• Existing alert.
• Manual smoke test.
• Dashboard check.
• Client workflow verification.
• Temporary heightened monitoring.