Rollback and Recovery

Purpose

This playbook defines how Forward Deployment teams restore service when a release causes production issues.

Standard

Every high-risk release must have a known rollback or recovery path before deployment.

Rollback does not always mean reverting code. It can mean:

• Redeploying the previous image or commit.
• Reverting a feature flag.
• Disabling an integration.
• Restoring a previous configuration.
• Running a recovery script.
• Fixing forward when rollback is riskier.

Decision Thresholds

Start rollback or recovery when:

• Core user flow is unavailable.
• Data integrity is at risk.
• Authentication is broken.
• External integration failure blocks the client workflow.
• AI output quality creates unacceptable business risk.
• Error rate or latency crosses the project’s defined threshold.
• The deployment owner cannot verify the release within the expected window.

Required Information

For high-risk releases, document:

• Last known good version.
• Rollback command or workflow.
• Data migration rollback notes, if relevant.
• Feature flag or config switch, if relevant.
• Health check to confirm recovery.
• Owner who can execute rollback.

Recovery Steps

1. Declare the issue in the internal project channel.
2. Assign a recovery owner.
3. Stop further deployments.
4. Confirm impact and affected users.
5. Choose rollback or fix-forward.
6. Execute recovery.
7. Verify health checks and affected workflow.
8. Update the release log or incident log.
9. Communicate status internally and to client if needed.

After Recovery

Create follow-up tasks for:

• Root cause investigation.
• Missing test coverage.
• Missing monitoring.
• Deployment workflow improvement.
• Documentation update.