Scan Archive and Cold Storage
MigrationFox retains every scan you run, indefinitely. To make that sustainable, the per-file detail of older scans is moved from the live Postgres database into Azure Blob cold storage once the scan has been idle past its retention window. The summary stays hot. The detail is one click away when you need it.
What stays in Postgres
Job name, status, counts (files, bytes, errors), started / completed / cancelled timestamps, owner, tenant, configuration, and error summary. Enough to render the job list and the job-detail summary with zero latency. Only the per-file rows (scan_items and file_records) move to cold storage.
What you’ll see
An archived scan looks identical to a non-archived one on the job list. Click into its JobDetail page and you’ll see one new element: a small gray Archived pill badge next to the job title in the header.
That badge is the only visible indicator. The job still shows its file count, byte count, start and completion times, any error summary, and all the other summary metadata. What it does not instantly show is the per-file list on the Files tab.
If you open the Files tab on an archived scan, you’ll see a card that reads something like “This scan’s file details have been moved to cold storage after its retention window. Click below to restore them.” with a single Restore file details button. Click it, wait a few seconds, and the file list is back and behaves normally. The badge clears.
Archive happens automatically. You don’t opt in and you don’t opt out. It runs once per day at 03:00 UTC and processes up to 100 eligible jobs per run.
Retention thresholds
Retention depends on the job’s terminal status. Different statuses idle for different durations before their per-file detail is moved. Only jobs in a terminal state are ever considered for archiving — in-flight scans are never touched.
| Status | Days idle before archive | Meaning |
|---|---|---|
CANCELLED | 1 day | You explicitly cancelled the scan. Almost never needed again. |
FAILED | 7 days | The scan terminated with an error. Usually triaged within a week. |
SCAN_COMPLETE | 14 days | Discovery finished but no transfer was run from it. |
COMPLETED | 14 days | Full migration ran and finished successfully. |
“Days idle” is measured from the timestamp the scan entered its terminal status, not from when it was first created. A scan that runs for three days and completes on day three is eligible for archive on day seventeen (completion + 14 days), not day seventeen-of-its-lifetime.
In-flight scans are never archived
Only jobs in CANCELLED, FAILED, SCAN_COMPLETE, or COMPLETED can archive. Anything in SCANNING, READY_TO_TRANSFER, TRANSFERRING, PAUSED, or any other active state is skipped by the archive cron. You can safely leave a scan paused for a month — it won’t archive while paused.
How to restore an archived scan
There is only one thing to do: click Restore file details on the Files tab of the archived scan. Typical restore completes in about 5 seconds. The longest restore we’ve measured is roughly 25 seconds, on a scan containing around a million rows.
Under the hood this hits POST /jobs/:id/restore, which fetches the blob from cold storage, decompresses it, re-inserts the rows into Postgres, clears the archive marker on the job, and returns. The page reloads and the Files tab shows the full list.
Once restored, the scan stays hot. It is eligible for archive again the next time it sits idle past its retention threshold, which means if you restore a COMPLETED scan today, it will be archived again after another 14 days of idleness. If you are actively working with a scan, just keep clicking around in it — any user-initiated interaction that touches scan_items counts as activity.
Restore is free
There is no charge for restoring, no limit on how many times you can restore the same scan, and no plan-level gating. Free tier, Growth, Pro, and Partner all restore identically. The cold-storage tier is an implementation detail, not a billable feature.
Resume, Retry, and other job actions
Resume and Retry both auto-rehydrate. You do not need to click Restore first.
If you click Resume on an archived scan, the worker detects the archive marker before anything else, fetches the blob, re-inserts the rows, clears the marker, and continues scanning. From your point of view this is transparent — you’ll see the scan go from CANCELLED or FAILED back into SCANNING or TRANSFERRING, with the rehydrate adding a few seconds of setup time on the first status update. You do not get a modal asking you to confirm the restore.
Retry behaves the same way. Under the hood it’s a Resume from the scan’s last known state, so the rehydrate path is identical.
Admin-triggered actions (bulk re-queue, revoke credential, etc.) also auto-rehydrate any archived scan they touch. In general, if an action needs the per-file rows, the system pulls them back for you and you never see the difference.
Admin controls
SUPER_ADMIN users have two additional endpoints and corresponding UI buttons for manual control of the archive lifecycle. These are intended for operations — catching up after a worker outage, forcing archive on a specific problem tenant, or testing the restore path.
Manually archive a single job
On the JobDetail page, SUPER_ADMIN users see an Archive now button in the admin tools panel. It calls POST /admin/jobs/:id/archive and forces the archive of that single job, regardless of its retention threshold. The job must still be in a terminal state.
Bulk archive
On the tenant admin page there is a Bulk archive modal. It calls POST /admin/archive-old-scans with a limit parameter, which runs the same cron logic you’d get at 03:00 UTC but on demand. Use this after bringing the worker back from a maintenance window to catch up on anything that should have archived while the worker was down.
Manual archive respects the terminal-state rule
Admin endpoints can force archive before the retention threshold, but they cannot archive a job that is still in an active state. If you need to stop a runaway scan and archive its rows, cancel it first, then archive.
Technical details
This section describes the implementation in enough detail to debug problems. Most users don’t need it.
Storage backend
- Service: Azure Blob Storage
- Storage Account:
migrationfoxarchive - Region: West US
- Tier: Cool
- Redundancy: LRS (locally redundant storage)
- Container:
migrationfox-scan-archives - Blob path format:
{tenantId}/{jobId}.json.gz
Blob payload
Each blob is a gzipped JSON object with two top-level arrays: scanItems (one entry per file or folder discovered by the scan) and fileRecords (one entry per file the migration attempted to transfer). All fields that the live Postgres table has are preserved, so the restore is lossless.
Compression ratio on real scan data is roughly 8x. A 300,000-row scan (mixed files and folders) compresses to about 14.7 MB of blob, which works out to around 50 bytes per row after gzip.
Credential storage
The Azure connection string for the archive account lives in the admin_config Postgres table, not in an environment variable. The worker reads it at scan-archive time. This is intentional: our worker runs on an Oracle Cloud container instance whose environment variables are immutable post-create, and we wanted to be able to rotate the Azure storage key without redeploying the worker.
Archive cron
- Schedule: daily at 03:00 UTC
- Queue:
governance(BullMQ on Redis) - Batch size: up to 100 eligible jobs per tick
- Ordering: oldest terminal-state timestamp first
Performance envelope
| Operation | Typical | Notes |
|---|---|---|
| Archive | 10–30 seconds / job | Dominated by Postgres read + gzip, not the blob PUT. |
| Restore (typical) | ~5 seconds | A few hundred thousand rows or fewer. |
| Restore (large) | ~25 seconds | Million-row scans; worker uses COPY for anything over 10k rows. |
| Blob size per row | ~50 bytes | After gzip. Raw JSON is closer to 420 bytes / row. |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Files tab shows empty list with no Restore button | Scan is not archived — it genuinely has 0 rows (e.g., zero-file scan of an empty source). | Check the summary — if the scan’s file count is 0, the Files tab is correctly empty. |
| Restore button appears but click does nothing | Browser is offline or the API is unreachable. | Check the browser network tab for the POST /jobs/:id/restore call. If it’s failing, the error surface on the page will show the reason (auth, network, 5xx). |
| Restore returns a 404 | The job’s archive marker is set, but the blob is missing from cold storage. Should be impossible in steady state. | Contact support. We can re-mark the job non-archived from the admin side, or if the original rows no longer exist, give you a clean report of what was lost. |
| Restore takes longer than 30 seconds | Scan has more than a million rows. Worker is using bulk COPY, which is fast but not instant. |
Wait. Million-plus row scans take up to ~25 seconds; anything beyond that is worth reporting. |
| Scan badge says “Archived” immediately after the scan completes | Not possible under the retention policy. File a bug report — this would be a real regression. | Contact support with the job ID. |
| Resume of an archived scan fails instantly | Rehydrate stage encountered an error — blob corrupt, network glitch mid-download, etc. | Retry the Resume. If the error reproduces, contact support with the job ID; we can restore manually from the admin endpoint. |
If restore repeatedly fails
Contact support at support@migrationfox.com with the job ID. We can trigger the restore from the admin side and, in the rare case of a missing blob, verify from our own backup path whether the data is recoverable. The underlying Azure Blob Cool tier has 99.9% SLA and LRS redundancy, so missing-blob events are vanishingly rare, but we want to know immediately if one happens.
FAQ
Does my scan data get deleted?
No. Archiving is not deletion. The scan record stays in Postgres and the per-file detail is preserved in Azure Blob cold storage, indefinitely, until you explicitly delete the job. You can restore it any time.
Do I pay for restored data?
No. Restore is free on every plan. There is no per-restore charge, no egress fee passed through, and no plan-level limit on how many times you can restore the same scan.
How long does restore take?
Typical restore is about 5 seconds. The longest we’ve measured is roughly 25 seconds for a scan containing around a million rows. If you’re consistently seeing longer times, contact support with the job ID.
Can I opt out of archiving?
No. Archiving is a platform-wide reliability feature, not a plan option. The retention thresholds are the same for every tenant. If you interact with a scan (click into it, resume, retry), its retention clock resets, so actively-used scans are never archived.
Will Resume or Retry fail because the scan is archived?
No. Resume and Retry detect the archive marker and rehydrate transparently before continuing. You will not be prompted, and the extra few seconds of setup happen before the first progress update.
What happens if I delete the job entirely?
Deleting the job deletes both the Postgres summary row and the cold-storage blob. That operation is irreversible. If you want to preserve the scan, keep the job and let it archive naturally; restore it only when you need it.
Is there any case where archived data would be lost?
Only if you delete the job yourself, or if the Azure Blob service lost the data entirely (the Cool tier SLA is 99.9% and the container uses LRS redundancy, so this is extremely unlikely). In normal operation, archived data is retained for as long as the job row exists.
Related
- Rollback preview — preview and undo a completed migration
- Revoke share links — lifecycle controls on Client Live View URLs
- Engineering blog: The 4am Postgres crash that shipped this architecture