training-software/docs/TRA-249-rollback-runbook.md

Rollback Runbook — Trainingssoftware

Use this runbook when a production release must be reverted. Decide on rollback within 15 minutes of a critical incident starting. Do not attempt to forward-fix if p95 latency > 2 s or error rate > 5% is sustained.

---

Decision criteria for rollback vs. forward-fix

Symptom	Recommended action
Error rate > 5% for > 5 min	Rollback immediately
p95 latency > 2 s for > 5 min	Rollback immediately
Healthcheck failing	Rollback immediately
Isolated 5xx on one endpoint	Consider forward-fix
Data inconsistency after migration	Rollback immediately + restore from backup

---

Step 1 — Communicate

1. Post in #incidents: "Initiating rollback of vX.Y.Z — reason: [brief description]"
2. Page the on-call engineer via PagerDuty if not already active.

Step 2 — Roll back the application

Container / Kubernetes rollback

# If using kubectl
kubectl rollout undo deployment/trainingssoftware-web
kubectl rollout status deployment/trainingssoftware-web

# Verify
kubectl get pods -l app=trainingssoftware-web

Docker Compose rollback (staging / single-host)

# Pull the previous tagged image
docker pull ghcr.io/company/trainingssoftware:vX.Y.Z-prev

# Update compose override to pin to old image, then:
docker compose up -d --no-deps web

Healthcheck verification

curl -sf https://app.example.com/healthz/ | jq .
# Expected: {"status": "ok", "checks": {"db": "ok", "cache": "ok"}}

Step 3 — Roll back migrations (if any were applied)

Only needed when the release contained schema migrations.

Check which migrations to revert

# On the deployment host, identify migrations introduced in the release
git diff vX.Y.Z-prev vX.Y.Z -- '*/migrations/*.py' | grep '^+++ ' | head -20

Revert migrations

# Revert each app to its previous state — replace app/0042 with the target state
python manage.py migrate <app_name> <previous_migration_number>

If the migration has no reverse (RunPython without reverse_code):

• Restore from the pre-deploy database backup (see Step 4).
• Do not attempt manual SQL rollback without DBA review.

Step 4 — Database restore (last resort)

Only if schema or data is irrecoverably corrupted.

# 1. Identify the pre-deploy dump
aws s3 ls s3://company-backups/trainingssoftware/ | grep "<deploy-date>"

# 2. Put the app in maintenance mode (return 503 from load balancer)

# 3. Restore to production (requires explicit confirmation)
export TARGET_DATABASE_URL="${DATABASE_URL}"
ops/scripts/restore.sh s3://company-backups/trainingssoftware/<dump-file>

# 4. Verify row counts (see backup-restore runbook)

# 5. Re-deploy the previous app image and re-run synthetic checks

Step 5 — Validate and close the incident

ops/scripts/synthetic-check.sh https://app.example.com

• Synthetic checks pass
• p95 latency back within SLO (≤ 500 ms)
• Error rate back within SLO (< 1%)
• No active critical Prometheus alerts
• Post incident summary in #incidents within 24 hours

---

Contacts

Role	Contact
On-call (PagerDuty)	Auto-escalation after 5 min
Database admin	DM via Slack #team-backend
Infrastructure	DM via Slack #team-infra