3d541d818a
- Add prometheus-client to base requirements; sentry-sdk to prod - api/metrics.py: define HTTP latency histogram, request/error counters, in-flight gauge - api/middleware.py: extend SecurityAuditMiddleware to observe all four Prometheus collectors per request; low-cardinality path_template label via URL resolver - api/views.py: /metrics/ endpoint (gated by METRICS_ENABLED setting) - api/urls.py: wire /metrics/ route - config/settings/prod.py: METRICS_ENABLED flag; optional Sentry SDK init via SENTRY_DSN env var - ops/prometheus/alerts.yml: Prometheus alert rules for p95 latency SLO (≤500 ms), error rate SLO (<1%), availability, and saturation - ops/prometheus/prometheus.yml: scrape config for app + blackbox healthcheck probe - ops/scripts/backup.sh: pg_dump → S3 STANDARD_IA with retention metadata - ops/scripts/restore.sh: pg_restore from S3 or local file with interactive confirmation guard - ops/scripts/synthetic-check.sh: post-deploy smoke test (healthz, metrics gate, schema, 404 shape) - docs/TRA-249-observability-slos.md: SLO table, PromQL reference queries, alert routing - docs/TRA-249-backup-restore.md: RPO/RTO targets, drill procedure, restore validation steps - docs/TRA-249-release-checklist.md: pre/post-deploy checklist - docs/TRA-249-rollback-runbook.md: decision matrix, app rollback, migration revert, DB restore path Co-Authored-By: Paperclip <noreply@paperclip.ing>
52 lines
1.7 KiB
Bash
Executable File
52 lines
1.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# backup.sh — PostgreSQL backup for Trainingssoftware
|
|
#
|
|
# Usage:
|
|
# DATABASE_URL=postgres://user:pass@host:5432/dbname \
|
|
# BACKUP_BUCKET=s3://my-bucket/backups \
|
|
# ops/scripts/backup.sh
|
|
#
|
|
# Environment variables:
|
|
# DATABASE_URL (required) — full postgres:// connection string
|
|
# BACKUP_BUCKET (optional) — s3:// prefix to upload the dump; if unset,
|
|
# the dump stays in /tmp only
|
|
# BACKUP_RETENTION_DAYS (optional, default 30) — S3 lifecycle handled externally;
|
|
# this value is embedded in the
|
|
# object metadata for auditing
|
|
set -euo pipefail
|
|
|
|
TIMESTAMP="$(date -u +%Y%m%dT%H%M%SZ)"
|
|
DUMP_FILE="/tmp/trainingssoftware-${TIMESTAMP}.dump"
|
|
RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-30}"
|
|
|
|
if [[ -z "${DATABASE_URL:-}" ]]; then
|
|
echo "ERROR: DATABASE_URL is not set." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "[backup] Starting PostgreSQL dump → ${DUMP_FILE}"
|
|
pg_dump \
|
|
--format=custom \
|
|
--compress=9 \
|
|
--no-acl \
|
|
--no-owner \
|
|
"${DATABASE_URL}" \
|
|
--file="${DUMP_FILE}"
|
|
|
|
DUMP_SIZE="$(du -sh "${DUMP_FILE}" | cut -f1)"
|
|
echo "[backup] Dump complete — size: ${DUMP_SIZE}"
|
|
|
|
if [[ -n "${BACKUP_BUCKET:-}" ]]; then
|
|
S3_KEY="${BACKUP_BUCKET%/}/trainingssoftware-${TIMESTAMP}.dump"
|
|
echo "[backup] Uploading → ${S3_KEY}"
|
|
aws s3 cp "${DUMP_FILE}" "${S3_KEY}" \
|
|
--metadata "retention_days=${RETENTION_DAYS},timestamp=${TIMESTAMP}" \
|
|
--storage-class STANDARD_IA
|
|
echo "[backup] Upload complete."
|
|
rm -f "${DUMP_FILE}"
|
|
else
|
|
echo "[backup] BACKUP_BUCKET not set — dump kept at ${DUMP_FILE}"
|
|
fi
|
|
|
|
echo "[backup] Done. timestamp=${TIMESTAMP} size=${DUMP_SIZE}"
|