Cookbook: resume CSV-to-Parquet batches

Commit small CSV-to-Parquet batches one at a time on mountable.io, so a fresh worker resumes without redoing or duplicating a batch.

A data worker often runs for a short time and then exits. If it converts a CSV to Parquet in batches, the next worker needs to know which batches are already done. A filename alone cannot say that, because a worker can stop after an upload and before anything records it.

mountable.io maintains this runnable example. It turns 60 made-up sales rows into 3 Parquet batches of 20 rows. Each run commits one batch and exits. A small JSON manifest records a batch only after its upload reads back with the right checksum. The data is synthetic and holds no customer data.

Setup

Use Python 3.11 or newer and uv. For the uploads, use an existing mountable.io account with a saved payment method, filesystem and API key. Keep MOUNTABLE_API_KEY in your private environment or secret manager.

git clone https://github.com/writeitai/mountable.git
cd mountable/examples/csv-parquet-batches
uv sync --locked
export FS_ID=your-filesystem-id
export RUN="demo-$(date +%s)"

Your job chooses FS_ID and RUN. Keep one run ID for every batch of a job, including the process that resumes it. Use a new run ID only for a different job: 1 to 40 lowercase letters, digits or hyphens, starting with a letter or digit.

Commit one batch at a time

uv run python etl.py run --filesystem "$FS_ID" --run "$RUN"

Each run builds one batch, uploads it as batches/<run>/batch-NNN.parquet (batch 000, 001, then 002) with a create-only write, reads it back, and compares the SHA-256. Only then does it add the batch to batches/<run>/manifest.json. The manifest update uses the ETag from the manifest read. If another writer changed the manifest first, the run stops with competing_writer and does not overwrite it.

Run the command three times to commit all 3 batches. A fourth run prints done and writes nothing.

Recover an interrupted batch

A worker can stop after the upload and before the manifest update. Give a second job its own run ID and try it there, while that job still has a batch left:

export RUN2="demo-crash-$(date +%s)"
uv run python etl.py run --filesystem "$FS_ID" --run "$RUN2" --crash-after-upload
uv run python etl.py run --filesystem "$FS_ID" --run "$RUN2"

The first command uploads the batch and stops. The batch is not recorded, so nothing counts it as done. The second run finds the uploaded file, checks it against the rebuilt batch, and prints adopted. It records the batch without uploading again, so no row appears twice. If the uploaded file does not match the rebuilt batch, the run stops with orphan_mismatch and records nothing.

Check it from a fresh process

uv run python etl.py check --filesystem "$FS_ID" --run "$RUN" --dir ./out

The check opens the filesystem read-only. It downloads the committed Parquet files, checks the columns (id,day,region,units,revenue_cents) and each file's checksum, and confirms every row id appears once. It prints match run=<run> batches=3 rows=60 unique_ids=60.

Clean up and costs

uv run python etl.py cleanup --filesystem "$FS_ID" --run "$RUN"
uv run python etl.py cleanup --filesystem "$FS_ID" --run "$RUN2"

Run the second line only if you tried the crash demo above.

Cleanup deletes the manifest, the batch files, and the run's directories when they are empty. At the USD usage prices, storage costs $0.20/GiB-month and downloads cost $0.02/GiB. Each batch file is a few kilobytes. Deleted data can stay billable until it is collected, so keep a retention policy and a filesystem spending cap on jobs that write files.

This recipe covers one writer at a time and small bounded batches. It does not claim transactions or cross-machine locks. For one closed dataset handed from Polars to DuckDB, see the Notebooks guide.

Offline checks

uv run pytest -q

The tests use a fake filesystem with ETags and real Polars. They cover the three-batch run, the crash between upload and manifest, a competing manifest writer, a changed upload that is never recorded, cleanup, read-only refusal, and bad run ids. No network or API key is needed.