Cookbook: a Hugging Face dataset after the worker exits

Save a tiny synthetic dataset with save_to_disk, upload that closed file set, and load the same rows from a fresh process.

A worker can build a Hugging Face Dataset, call save_to_disk, and then exit. The next process does not have that directory.

mountable.io is a persistent filesystem for AI agents. This recipe uploads the closed one-shard files before it writes a completion record. A later process checks those files and then calls load_from_disk.

mountable.io maintains the runnable example, separately from Hugging Face. The rows are synthetic. They hold no customer data. This is not a Hub upload, a zip archive, or a remote filesystem adapter, and it does not need a Hugging Face account.

Prerequisites

Use an existing mountable.io account with a saved payment method, an existing filesystem and an API key that can access it. Storage and file operations follow the usage prices. The example creates no account, filesystem or credits.

You also need Python 3.11 or newer and uv. Set MOUNTABLE_API_KEY in your private environment or secret manager. Never put a key in a command argument.

git clone https://github.com/writeitai/mountable.git
cd mountable/examples/huggingface-datasets
uv sync --locked
export MOUNTABLE_FILESYSTEM_ID="your-filesystem-id"
export DATASET_STORE=mountable

The example pins datasets 5.1.0 and the Python SDK. It calls Dataset.save_to_disk with one shard and load_from_disk on a local directory. It does not mount the filesystem. For one filesystem and a hard monthly spend cap per end user, follow the capped workspace guide. This recipe uses one existing filesystem and a path your backend chooses.

The dataset belongs to the fixture user alice. Your own worker would choose its real user. It would still choose the filesystem and the path, and it would not take either from the dataset arguments.

Write the dataset

uv run python dataset.py write

A successful run prints the dataset id, the row count, the feature label, the rows checksum, the file count and the byte count:

dataset_id=a1b2c3d4e5f60789
row_count=2
features=item:string,count:int64
rows_sha256=<sha256 of the two rows>
files=3
bytes=<size>

The dataset id is 16 lowercase hex characters. It is not a filesystem id and it is not a path. The command saves two rows, alpha / 1 and beta / 2, with features item:string and count:int64. With one shard, save_to_disk writes these files:

data-00000-of-00001.arrow
dataset_info.json
state.json

The command uploads those three files under datasets/alice/<id>/, reads each one back, and only then writes complete.json. Each write uses If-None-Match: *, so an existing file is left as it is.

If an upload fails, the command exits without a completion record. A later read of that id reports the dataset as incomplete.

The filesystem id stays in the environment. The worker passes the dataset id only.

Read it from a fresh process

Start a new process with the same environment. Pass the id the writer printed:

uv run python dataset.py read --dataset-id "$ID"

This process checks the completion record, then each file's size and checksum. A missing file, a changed file, or a completion record that does not list the known files stops the read before load_from_disk. When the checks pass, it loads the local copy and checks the rows and features. A matching dataset prints one line: match, the path, the row count, the feature label, the rows checksum and the file count.

A second create of the same id does not replace the arrow file:

uv run python dataset.py overwrite --dataset-id "$ID"

That command fails, and the first files stay as they are. A read-only handle cannot write them either:

uv run python dataset.py read-only-write --dataset-id "$ID"

One command runs that check for you. It writes the dataset, reads it from a new process, rejects a replacement, rejects a read-only write, and deletes the files:

uv run python dataset.py live

It prints match and the fields above. It does not print the API key.

A 402 with filesystem_spending_cap_reached stops that filesystem's billable mountable.io work. Other filesystems keep working.

The id the writer returns is the handoff. The bytes stay in the known files. This recipe does not upload the dataset to the Hub, and it does not pass a remote filesystem to save_to_disk.

Clean up

uv run python dataset.py cleanup --dataset-id "$ID"

This deletes the completion record and the three dataset files. A file that is already gone is left gone. Against mountable.io it also removes the empty parent directories for that dataset.

Offline checks

From the example directory:

uv run pytest -q

The tests call the real save_to_disk and load_from_disk with a fake store, and they also start two local processes over a temporary directory. They make no paid cloud calls. They check the rows, features, checksums, a failed upload, a changed file, a missing file, and that a second create leaves the first bytes in place.

Sources: Dataset, checked against datasets 5.1.0 on 2026-10-27, and the Python SDK.