Owner: Lola (Ops) · Department: Operations · Status: Live · Version: 1.0
Effective Date: 2026-05-13 · Last Reviewed: 2026-06-13 · Next Review Date: 2026-09-13
Source of Truth: this page · Maturity: 4 (Operational)
| Field | Value |
|---|---|
| Severity | P2 - High |
| PagerDuty Service | HempDash Orders |
| Escalation Policy | Operations Standard |
| Response SLA | 15 minutes |
| Owner | On-Call Engineer / Ops |
| Alert Name | WARNING-BIZ-ORDERS-STUCK |
An order that has remained in a non-terminal state longer than its SLA:
| Status | SLA | Meaning |
|---|---|---|
new |
15 min | Vendor hasn't acknowledged |
accepted |
30 min | Not packed yet |
packed |
60 min | Not dispatched to courier |
shipped |
240 min | Not delivered |
Terminal states (delivered, cancelled, rejected, refunded) are never "stuck."
The detect_stuck_orders Celery task runs every 5 minutes and emits:
| Metric | Description |
|---|---|
stuck_orders_current{status="new"} |
Gauge: orders stuck in new |
stuck_orders_current{status="accepted"} |
Gauge: orders stuck in accepted |
stuck_orders_current{status="packed"} |
Gauge: orders stuck in packed |
stuck_orders_current{status="shipped"} |
Gauge: orders stuck in shipped |
Alert thresholds:
pending -> new -> accepted -> packed -> shipped -> delivered
-> refunded
new -> rejected
new -> cancelled
accepted -> cancelled
packed -> cancelled
An order cannot skip states. If an order is in new, it must go through accepted before packed.
-- Orders stuck beyond SLA
SELECT o.order_id, o.status, o.vendor_id, o.user_id,
o.placed_at, NOW() - o.placed_at AS age,
v.name AS vendor_name
FROM orders o
LEFT JOIN vendors v ON o.vendor_id = v.id
WHERE o.status IN ('new', 'accepted', 'packed', 'shipped')
AND (
(o.status = 'new' AND o.placed_at < NOW() - INTERVAL '15 minutes')
OR (o.status = 'accepted' AND o.placed_at < NOW() - INTERVAL '30 minutes')
OR (o.status = 'packed' AND o.placed_at < NOW() - INTERVAL '60 minutes')
OR (o.status = 'shipped' AND o.placed_at < NOW() - INTERVAL '240 minutes')
)
ORDER BY o.placed_at ASC;
| Stuck in | Common Causes |
|---|---|
new |
Vendor not logged in, vendor app down, push notification failed |
accepted |
Kitchen/packing delay, short-staffed |
packed |
No courier available, dispatch system failure |
shipped |
Driver issue, GPS not updating, app crash |
-- Is the vendor online?
SELECT id, name, is_active, last_seen_at
FROM vendors
WHERE id = '<vendor_id>';
-- Is there a courier assigned?
SELECT c.id, c.name, c.status, c.last_location_at
FROM couriers c
JOIN order_assignments oa ON c.id = oa.courier_id
WHERE oa.order_id = '<order_id>';
new (vendor hasn't seen it)accepted or packedpacked/shippedshippeddeliveredThe order status machine in order_status_machine.py enforces valid transitions. Skipping states (e.g., new directly to shipped) will return HTTP 409.
| Time | Action |
|---|---|
| 0 min | On-call identifies stuck orders and root cause |
| 15 min | Contact vendor/courier for manual resolution |
| 30 min | If systemic (multiple vendors/orders), check platform health |
| 45 min | Escalate to Jonathan if customer impact is growing |
| 60 min | Consider cancelling and refunding affected orders |
detect_stuck_orders task is running correctlyapp/services/order_status_machine.pyapp/tasks/stuck_order_detector.pyapp/models/order.pyPlaybooks index · Severity Matrix · Alert → Owner Map · Home