Owner: Robert Taylor (Eng) · Department: Engineering · Status: Live · Version: 1.0
Effective Date: 2026-05-13 · Last Reviewed: 2026-06-13 · Next Review Date: 2026-09-13
Source of Truth: this page · Maturity: 4 (Operational)
| Field | Value |
|---|---|
| Severity | P1 - Critical |
| PagerDuty Service | HempDash Payments |
| Escalation Policy | Payment Critical |
| Response SLA | 15 minutes |
| Owner | On-Call Engineer + Finance |
| Alert Name | WARNING-BIZ-FINANCE-PAYOUT_ANOMALY |
A vendor or courier may have received the same payout twice (or a payout for an amount they shouldn't have received). This is a financial incident — every minute matters because recovery becomes harder once funds are withdrawn.
| Source | Signal |
|---|---|
| Grafana | WARNING-BIZ-FINANCE-PAYOUT_ANOMALY fires on flagged payouts |
| Reconciliation Task | reconcile_daily_payouts runs at 2 AM UTC — check results |
| Manual Report | Vendor/courier reports receiving unexpected payment |
| ERPNext | Issue ticket auto-created on payout failure |
-- Find potential duplicates: same order paid out more than once
SELECT order_id, provider, COUNT(*) as payout_count,
SUM(amount) as total_paid
FROM finance_ledger
WHERE ledger_type = 'PAYOUT'
AND status IN ('PAYOUT_COMPLETED', 'PAYOUT_PENDING')
GROUP BY order_id, provider
HAVING COUNT(*) > 1
ORDER BY total_paid DESC;
-- Were different idempotency keys used for the same order?
SELECT fl.order_id, fl.amount, fl.status, fl.created_at,
fik.idempotency_key
FROM finance_ledger fl
LEFT JOIN finance_idempotency_key fik ON fl.id = fik.ledger_id
WHERE fl.order_id = '<order_id>'
AND fl.ledger_type = 'PAYOUT'
ORDER BY fl.created_at;
If the same order has multiple ledger entries with different idempotency keys, the dedup guard was bypassed — likely a code bug or race condition.
-- Payout request history for this vendor/courier
SELECT id, status, amount, payout_method, created_at, completed_at
FROM payout_requests
WHERE vendor_id = '<vendor_id>' -- or courier_id
AND created_at > NOW() - INTERVAL '7 days'
ORDER BY created_at DESC;
-- Was this part of the nightly settlement batch?
SELECT * FROM settlement_batches
WHERE created_at > NOW() - INTERVAL '48 hours'
ORDER BY created_at DESC;
The nightly_vendor_settlement_batch runs at midnight UTC. If a manual payout was triggered AND the nightly batch also processed it, that's a common duplicate pattern.
order_idFreeze further payouts to the affected vendor/courier:
UPDATE payout_provider_account
SET is_active = false
WHERE vendor_id = '<vendor_id>';
Calculate the overpayment amount:
SELECT SUM(amount) - expected_amount AS overpayment
FROM finance_ledger
WHERE order_id = '<order_id>'
AND ledger_type = 'PAYOUT'
AND status = 'PAYOUT_COMPLETED';
Contact the vendor/courier — inform them of the duplicate and arrange recovery:
Create adjustment ledger entry:
INSERT INTO finance_ledger (order_id, provider, ledger_type, amount, status, notes)
VALUES ('<order_id>', '<provider>', 'ADJUSTMENT', -<overpayment>, 'COMPLETED',
'Duplicate payout recovery - INC-PAY-002');
Re-enable payouts after recovery is confirmed:
UPDATE payout_provider_account
SET is_active = true
WHERE vendor_id = '<vendor_id>';
| Cause | Fix |
|---|---|
| Race condition in settlement + manual payout | Add mutex lock on payout creation per order |
| Idempotency key not checked | Verify finance_idempotency_key constraint is enforced |
| Retry created duplicate | Check PayoutService.max_retries and backoff config |
| ERPNext sync triggered re-payout | Review _sync_payout_failure_to_erpnext() logic |
| Time | Action |
|---|---|
| 0 min | On-call investigates — do not process more payouts until confirmed |
| 15 min | Confirm whether duplicate is real or false alarm |
| 30 min | If real, freeze affected accounts and notify Jonathan |
| 60 min | Contact vendor/courier for recovery arrangement |
| 24 hrs | Verify reconciliation task caught the discrepancy |
reconcile_daily_payouts task — did it catch this?app/services/payout_service.pyapp/tasks/reconcile_payouts.pyapp/tasks/scheduled_payouts.pyapp/models/payout_requests.pyapp/models/finance_ledger.pyPlaybooks index · Payment Architecture · Severity Matrix · Home