Owner: Robert Taylor (Eng) · Department: Engineering · Status: Live · Version: 1.0
Effective Date: 2026-05-13 · Last Reviewed: 2026-06-13 · Next Review Date: 2026-09-13
Source of Truth: this page · Maturity: 4 (Operational)
| Field | Value |
|---|---|
| Severity | P1-P3 (depends on what broke) |
| PagerDuty Service | HempDash Platform |
| Escalation Policy | Platform Standard |
| Response SLA | 5 minutes (P1), 15 minutes (P2) |
| Owner | On-Call Engineer |
| Alert Name | Various (deployment-triggered failures) |
Rollback when a deployment causes:
DO NOT rollback for:
| Service | Branch | Health Check | Domain |
|---|---|---|---|
| Backend API | main |
GET /health |
api.gethempdash.com |
| Celery Worker | main |
Process alive | (internal) |
| Automation | master |
GET /api/health |
automation.gethempdash.com |
| ETA Service | main |
GET /health |
(internal) |
Frontend services (Netlify):
| Service | Branch | Domain |
|---|---|---|
| Customer App | main |
app.gethempdash.com |
| Vendor Portal | main |
store.gethempdash.com |
| Courier Portal | main |
drive.gethempdash.com |
| Ops Dashboard | main |
ops.gethempdash.com |
| Website | main |
web.gethempdash.com |
Railway will redeploy the previous container image. This takes ~30-60 seconds.
# List recent deployments
railway deployments list --service <service-name>
# Rollback to a specific deployment
railway rollback --service <service-name> --deployment <deployment-id>
Railway auto-deploys on push to the tracked branch. A dashboard rollback will be overwritten on the next push. To make the rollback permanent:
# Revert the bad commit
git revert <bad-commit-sha>
git push origin main
This triggers a new deployment with the reverted code.
netlify deploy --prod --dir=<build-dir> --site=<site-id>
CRITICAL: If the bad deployment included a database migration, rollback is more complex.
# Connect to the database
railway connect postgres
# Check migration history
SELECT * FROM alembic_version ORDER BY version_num DESC LIMIT 5;
# Downgrade one migration
railway run alembic downgrade -1
# Or downgrade to a specific version
railway run alembic downgrade <version>
Warning: Destructive migrations (DROP COLUMN, DROP TABLE) cannot be reverted. Check the migration file before downgrading.
The migration uses advisory lock ID 999999 to prevent concurrent runs.
# Backend
curl -s https://api.gethempdash.com/health | jq '.status'
# Automation
curl -s https://automation.gethempdash.com/api/health | jq '.status'
# Backend logs (last 5 minutes)
railway logs --service hempdash-backend --limit 100
# Worker logs
railway logs --service hempdash-backend-worker --limit 50
Before rolling back, check if the broken feature can be disabled instead:
| Flag | Purpose | Location |
|---|---|---|
ENABLE_PAYMENTCLOUD |
PaymentCloud payment processing | Backend env |
ENABLE_SEZZLE |
Sezzle BNPL | Backend env |
ENABLE_VIP |
VIP/loyalty features | Backend env |
ENABLE_SPRINGBIG |
Springbig integration | Backend env |
LOUD_DEPT_ENABLED |
LOUD DEPT features | Automation env |
Toggle in Doppler — Railway auto-redeploys with the new value.
| Time | Action |
|---|---|
| 0 min | Identify the bad deployment and initiate rollback |
| 5 min | Verify rollback succeeded via health check |
| 10 min | Run smoke tests on critical flows |
| 15 min | If rollback doesn't fix it, check if migration is involved |
| 30 min | Escalate to Jonathan if revenue-impacting and still unresolved |
railway.toml, Dockerfilescripts/migrate.shPlaybooks index · Release Governance SOP · Severity Matrix · Home