104 lines
2.8 KiB
Markdown
104 lines
2.8 KiB
Markdown
# Order Backup CLI — Design Spec
|
|
|
|
## Overview
|
|
|
|
CLI tool to back up completed (Recived) orders from production Azure Table by time range to CSV files, then optionally delete backed-up records. Operates on production storage, so all operations are throttled to minimize performance impact.
|
|
|
|
## Scale
|
|
|
|
- ~12,000 orders/day
|
|
- ~6.5M total rows (1.5 years)
|
|
- Backup targets: Recived orders only (state=1)
|
|
|
|
## Configuration
|
|
|
|
File: `~/.config/order-bak/config.toml`
|
|
|
|
```toml
|
|
[azure]
|
|
connection_string = "DefaultEndpointsProtocol=..."
|
|
|
|
[scan]
|
|
page_size = 500
|
|
delay_ms = 200
|
|
scan_dir = "./scan"
|
|
|
|
[delete]
|
|
batch_size = 100
|
|
delay_ms = 300
|
|
```
|
|
|
|
## Commands
|
|
|
|
### `scan`
|
|
|
|
```
|
|
order-bak scan
|
|
```
|
|
|
|
Full table scan with continuation token pagination. No server-side filter — save all orders. Save results to `scan_dir/scan-<date>.csv`.
|
|
|
|
- Single-threaded, sequential with configurable page size and delay
|
|
- Default: 500 rows/page, 200ms inter-page delay
|
|
- Estimated time: ~43 minutes for 6.5M rows
|
|
|
|
### `backup`
|
|
|
|
```
|
|
order-bak backup --start 2025-01-01 --end 2025-06-30
|
|
```
|
|
|
|
Read from local scan CSV file (default: latest `scan-*.csv` in `scan_dir`, overridable with `--scan <path>`), filter by `state == Recived` and `OrderTime` in `[start, end)`, output one CSV per date under `backups/`.
|
|
|
|
- Pure local operation, no Azure access
|
|
- Print summary after completion: total matched, per-day distribution
|
|
|
|
### `delete`
|
|
|
|
```
|
|
order-bak delete --csv backups/2025-01-01.csv
|
|
```
|
|
|
|
Read PartitionKey and RowKey from CSV. Group by PartitionKey, batch delete using Azure Table Transactions (max 100 per transaction, same partition). Print record count and require user confirmation before deleting.
|
|
|
|
## Output Directory Structure
|
|
|
|
```
|
|
scan/
|
|
└── scan-2026-04-21.csv # scan output (all orders)
|
|
|
|
backups/
|
|
├── 2025-01-01.csv # backup output (per-date CSV)
|
|
├── 2025-01-02.csv
|
|
└── ...
|
|
```
|
|
|
|
All CSV files share the same columns: `PartitionKey, RowKey, OrderTime, ReceiveTime, revenue, platformOrder, prodCount, pfId, prodId, prodPrice, state, ver, customData`
|
|
|
|
## Azure Table Schema (reference)
|
|
|
|
Table name: `order`
|
|
|
|
| Field | Type | Notes |
|
|
|---|---|---|
|
|
| PartitionKey | string | User ID (pfId) |
|
|
| RowKey | string | Order ID |
|
|
| OrderTime | DateTime | Order creation time (UTC) |
|
|
| ReceiveTime | DateTime | Confirmation time (UTC) |
|
|
| revenue | double | Charged RMB amount |
|
|
| platformOrder | string | Platform order ID |
|
|
| prodCount | int | Product count |
|
|
| pfId | string | Platform ID |
|
|
| prodId | string | Product ID |
|
|
| prodPrice | double | Product price |
|
|
| state | byte | 0=Payed, 1=Recived |
|
|
| ver | int | Schema version |
|
|
| customData | string | JSON blob |
|
|
|
|
## Tech Stack
|
|
|
|
- Python, managed with `uv` (installable via `uv tool`)
|
|
- Azure Data Tables SDK (`azure-data-tables`)
|
|
- TOML config via `tomllib` (stdlib in Python 3.11+)
|
|
- CSV output via `csv` stdlib module
|