Production Incident

Purpose

This playbook defines how the team responds to production incidents.

Severity Levels

Severity	Meaning	Example
SEV1	Major production outage or data risk.	Client workflow unusable, data corruption risk, auth outage.
SEV2	Significant degradation with workaround.	Core workflow slow or partially broken.
SEV3	Limited impact or non-critical issue.	Isolated bug, minor integration failure.

Required Roles

Role	Responsibility
Incident Commander	Coordinates response and decisions. Usually Project Technical DRI or available senior engineer.
Technical Responder	Investigates and fixes the issue.
Communications Owner	Handles client-facing updates. Usually Deployment Strategist.
Timeline Owner	Records timestamps, actions, and decisions. Can be the Incident Commander on small incidents.

On small teams, one person can hold multiple roles, but the Incident Commander and Communications Owner should be explicit.

Incident Workflow

1. Declare incident and severity.
2. Assign roles.
3. Create or update the Notion Incident Log entry.
4. Confirm client impact.
5. Stabilize the system.
6. Communicate status internally.
7. Communicate to client if client impact exists.
8. Resolve or mitigate.
9. Confirm recovery.
10. Create follow-up tasks.

Timeline Logging

Record:

• Detection time.
• Impact start time, if known.
• First response time.
• Key investigation findings.
• Mitigation time.
• Recovery time.
• Client updates sent.
• Follow-up owners.

Communication

Internal updates should include:

• Current impact.
• Current hypothesis.
• Owner.
• Next update time.

Client updates should be factual and concise:

• What is affected.
• What Calibrax is doing.
• Whether there is a workaround.
• When the next update will be sent.

Do not speculate in client updates.