AR Controls ERP OP-DEXT-B

Finance Centre · Invoice Compliance

Invoice Backup
& Recovery

Proving HMRC-grade durability for every invoice
processed through the Finance Centre

AR Controls ERP  ·  June 2026  ·  Concept Specification

The Problem

Invoices Can Go Missing — Again

A previous server cleanup wiped locally-stored invoice PDFs. The system recovered — but only by luck. Three active failure paths remain today.

⚡

Risk 1 — DO Spaces bypass: When DigitalOcean Spaces is configured, invoices upload there and the SharePoint backup never fires. A billing lapse or outage loses files permanently.

🔇

Risk 2 — Silent failure: SharePoint upload is fire-and-forget (asyncio.create_task()). Failures are logged but never surfaced — no retry, no alert, no status on the receipt record.

📁

Risk 3 — DB backup ≠ PDF backup: The daily encrypted MongoDB backup covers document metadata only. PDF files are excluded. Restoring the database returns broken receipt records with missing attachments.

Current State

What Exists Today

✓ In Place

  • Invoice ingest pipeline running (email + manual upload)
  • SharePoint per-ingest backup code exists (storage_service.py)
  • Daily MongoDB backup → ERP Backups/ (encrypted)
  • SharePoint upload_receipt() stores to ERP Receipts/{YYYY}/{MM}/
  • Restore endpoint exists (POST /api/backup/restore)

✗ Unverified / Missing

  • No proof backup runs on every ingest
  • No backup status field on receipt documents
  • No UI visibility into backup health
  • No PDF restore endpoint
  • No wipe+recover drill ever run end-to-end

⚠ HMRC requires 6-year invoice retention.  Until the backup chain is proven end-to-end, this obligation is not demonstrably met.

Gap Analysis

Where the Chain Breaks

Path A — DigitalOcean Spaces configured (production)

Invoice
arrives
→
DO Spaces
primary upload ✓
→
SharePoint
SKIPPED ✗
→
MongoDB
metadata only ✗

Path B — Local storage (fallback)

Invoice
arrives
→
Local FS
uploads/receipts/ ✓
→
SharePoint
best-effort, no status ⚠
→
MongoDB
metadata ✓

Bottom line: In neither path is it currently guaranteed that every invoice PDF is safely in SharePoint. The daily DB backup exists but does not include PDF bytes — only metadata. A server wipe today would recover database records but leave PDFs permanently lost.

The Solution

Three-Part Hardening

1
Close the DO_SPACES bypass
After every successful upload (Spaces or local), schedule a SharePoint shadow copy regardless of primary storage path. Modify storage_service.py:upload_file() to always call _backup().
2
Write backup status to receipt document
Add sharepoint_backup_status (pending / success / failed), sharepoint_backup_at, and sharepoint_backup_path to each receipt record. Failures surface in UI — no more silent drops.
3
Add PDF restore endpoint & Finance Centre UI
New POST /api/receipts/restore-from-sharepoint walks the receipts collection, downloads missing PDFs from ERP Receipts/{YYYY}/{MM}/. Finance Centre Settings gains a Backup & Recovery tab with health stats and restore wizard.
4
Run and document the wipe+recover drill on DEV
Delete local files, restore from SharePoint, confirm Finance Centre renders all receipts. Transcript written to evidence/OP-DEXT-B/.

UI Design

Finance Centre — New Screens

Finance Centre → Settings → Backup & Recovery
SharePoint Connected
✓ 839 Backed Up ✗ 8 Failed ○ 0 Pending

Last backup: 15 Jun 2026 14:32 · Last DB backup: 15 Jun 2026 02:01 ✓

Failed Backups (8)

#1042 Travis Perkins — SharePoint timeout  [Retry]

#1037 Würth UK — Token refresh failed  [Retry]

Receipt #1041 — Travis Perkins

Travis Perkins

INV-2026-0041 · 14 Jun 2026

Pending Review ✓ SharePoint Not Exported

Backed up · 14 Jun 2026 at 14:32
Path: ERP Receipts/2026/06/abc123.pdf

📄 travis_perkins_14jun.pdf

Disaster Recovery

Wipe + Recover Drill

🗑️

Step 1 — Wipe

Delete local files in uploads/receipts/ and storage/email_attachments/ on DEV only

🗄️

Step 2 — Restore DB

Upload latest .json.enc via POST /api/backup/restore. Receipt records return to MongoDB.

☁️

Step 3 — Restore PDFs

Call POST /api/receipts/restore-from-sharepoint. Downloads PDFs from ERP Receipts/.

✅

Step 4 — Verify

Finance Centre renders all receipts. Same IDs, amounts, and PDF attachments as before the wipe.

⚠ Safety constraint:  All destructive steps run on DEV (100.68.36.49:3080) only. Production (erp.ar-controls.co.uk) is never touched. Drill transcript written to evidence/OP-DEXT-B/restore-drill-transcript.md.

Implementation

Technical Approach

Files Modified

FileChange
storage_service.py Plug DO_SPACES bypass; write status back to receipt doc
receipts.py New restore endpoint; surface backup status in responses
sharepoint_service.py Fix error paths; return structured result with path
ReceiptsSettings.jsx Add "Backup & Recovery" tab
ReceiptDetail.jsx Add SharePoint backup status badge

Data Model Change

// receipts MongoDB document
sharepoint_backup_status:
  "pending" | "success" | "failed"
sharepoint_backup_at: ISODate
sharepoint_backup_path: String
sharepoint_backup_error: String

Stack

FastAPI MongoDB / Motor Microsoft Graph API MSAL OAuth 2.0 React + JSX Fernet Encryption

Done When…

Success Criteria

  • ✓ Code audit: SharePoint backup fires on every ingest path (DO Spaces AND local)
  • ✓ Live trace on DEV: logs confirm SharePoint upload for a real test invoice
  • ✓ Receipt document shows sharepoint_backup_status: success
  • ✓ Wipe+recover drill: 100% of invoices restored from SharePoint on DEV
  • ✓ Finance Centre renders all receipts post-restore (browser-verified)
  • ✓ Zero silent backup failures — errors surface in UI and logs at ERROR level
  • ✓ Backup & Recovery tab live in Finance Centre Settings
  • ✓ Restore drill transcript written (evidence/OP-DEXT-B/)
  • ✓ VERIFICATION.md complete with audit + trace + drill evidence
  • ✓ .orch-done.json written after end-to-end proof
100%
Receipts with SharePoint shadow copy
0
Silent backup failures
<15m
Full receipt file restore time
6yr
HMRC retention — demonstrable

Delivery Plan

Next Steps

▶ This Slice — OP-DEXT-B Audit

  • 1. Audit code path in storage_service.py
  • 2. Live trace on DEV — trigger test ingest, confirm logs
  • 3. Wipe local files (DEV only), restore from SharePoint
  • 4. Screenshot Finance Centre post-restore
  • 5. Write evidence/OP-DEXT-B/VERIFICATION.md
  • 6. Write .orch-done.json

◯ Later Slices — OP-DEXT-B Full Scope

  • OCR upgrade — measure accuracy on 20 real invoices, improve model if <95%
  • Multi-page auto-detect — split PDFs at pre-processing, no manual split
  • Fuzzy PO match — score thresholds, flag-for-review at 0.6–0.9
  • 50-invoice soak test — end-to-end headless browser verification
  • Runbook — brain/runbooks/dext-recovery.md

Questions before implementation → see concept.md §Open Questions  (SharePoint DEV credentials, DO_SPACES config, backup timing preference)

1 / 10