Resume a CSV-to-Parquet Job After the Worker Restarts

A batch worker exits after one Parquet batch. A verified manifest on mountable.io lets the next worker resume without redoing or duplicating a batch.

Say you run a small batch job in a worker. It turns a CSV into Parquet files, 20 rows at a time.

The worker finishes the first batch and exits. The next worker starts with no memory of the first one.

If it guesses wrong, it redoes finished batches or writes the same rows twice. This recipe writes that progress on mountable.io, a persistent filesystem for AI agents.

Redoing work is the easy fix, and the extra files stay on paid storage until you delete them. Trusting a filename is cheaper and riskier.

A worker can upload a file and stop before anything records that it finished. Each closed Parquet batch needs a record the next worker can trust.

That record is one small manifest. It names the batches that are done.

Commit one batch, then exit

The example builds 60 made-up sales rows and splits them into 3 batches. One run commits one batch:

fs.write(path, parquet_bytes, if_none_match=True)
back = fs.read(path)
assert sha256(back.data) == digest  # only then update the manifest

The upload is create-only, so a retried run cannot replace a batch by mistake. The run reads the file back and compares its SHA-256.

Only a verified file goes into the manifest. The manifest update uses the ETag from the manifest read, so the write fails if that file changed in between.

The run that stops halfway

The hard case is a worker that uploads its batch and stops before the manifest update. The file is there. Nothing says it counts.

The next run handles this on purpose. It finds the uploaded file before it uploads anything.

It rebuilds the same batch from the CSV and compares checksums. A matching file is recorded in the manifest without a second upload.

No row appears twice. The interrupted batch stops being a guess.

A second writer stops the run

Two workers can read the same manifest and both try to record a batch. The second one's ETag no longer matches, so its manifest write fails with competing_writer.

It stops instead of overwriting the first worker's record. The next step is boring on purpose.

Run the commit command again. That command reads the manifest and continues from the batches it lists.

Check it from a fresh process

A separate check opens the filesystem read-only. It downloads the batches the manifest names and counts them:

match run=demo-1 batches=3 rows=60 unique_ids=60

The check compares each file's checksum and confirms every row id appears once. Batches the manifest does not name are not counted, so an unfinished upload never reads as done work.

Run it yourself

Run the commit command three times and let each process exit. Then run the read-only check from another shell.

Delete the run's files with the cleanup command when you finish. The cookbook has the full commands, the crash demo, and the offline tests.

The Notebooks guide shows the same filesystem inside Kaggle, Colab, Deepnote, Hugging Face Spaces and Marimo.