Haystack: pipeline configuration across workers

Save a trusted Haystack pipeline as YAML on mountable.io, then load it locally and repeat the same run in a fresh worker.

mountable.io is a persistent filesystem for AI agents. A Haystack job can serialize its component settings and connections to a file before its worker exits. The next worker downloads that file and runs the pipeline locally.

mountable.io maintains this community recipe, separately from Haystack. It uses Haystack's pipeline serialization API with two built-in local components. The YAML holds configuration, not documents, an index, running state or execution checkpoints. Each process supplies the synthetic documents again.

Set up

Use Python 3.11 or newer and uv. The lockfile pins haystack-ai==3.4.1 and mountable-sdk==0.3.0. DocumentCleaner removes extra whitespace, then DocumentSplitter splits by seven words with no overlap. This word-based fixture needs no tokenizer download, model service or provider key. The example disables Haystack telemetry before importing it.

You need an existing mountable.io account with a saved payment method, a filesystem and an API key allowed to use it. Set MOUNTABLE_API_KEY in your private environment. Your trusted backend chooses the filesystem and run id; do not take either from a model or an untrusted caller.

git clone https://github.com/writeitai/mountable.git
cd mountable/examples/haystack-pipeline
uv sync --locked
export FS_ID="your-filesystem-id"
export RUN="pipeline1"

Choose a new run id for every publish: 1–40 lowercase letters, digits or hyphens, starting with a letter or digit. The file lives at haystack/<run>.yaml.

Save the pipeline and run it in a fresh process

uv run python handoff.py publish --filesystem "$FS_ID" --run "$RUN"

The publisher runs the pipeline, calls Pipeline.dump() on a local YAML file, closes it and uploads its 977 bytes with a create-only write. It prints sha256=<64-character digest>, followed by:

["The synthetic observatory has a blue dome.", "The telescope opens at dusk."]

Pass that digest through trusted job metadata outside the shared prefix. Then start a separate process:

export SHA256="paste-the-digest-from-publish"
uv run python handoff.py run --filesystem "$FS_ID" --run "$RUN" --sha256 "$SHA256"

The new process uses a read-only SDK handle. It verifies the downloaded bytes before calling Pipeline.load(), supplies the same synthetic input and prints the same two strings. This flow passed against production storage. A repeated publish returned 412 precondition_failed, a write through the read-only handle returned 403 read_only, and changed YAML was refused before loading.

A second publish never replaces the first file. If a request times out, read and inspect the owned path before another write. An upload may have succeeded despite a lost response; clean up and choose a new run id when starting again.

Keep configuration trusted

Only load pipeline files from your own trusted publisher. Loading YAML can instantiate the classes it names. A fingerprint detects changed bytes; it does not prove authorship or make an arbitrary downloaded pipeline safe.

The reader checks SHA-256, a 16 KiB maximum and byte-for-byte equality with this example's locally defined configuration. That last check deliberately limits this runnable fixture to its two known components. It is not a general loader for other pipeline definitions. Review components and configuration when adapting it for your own job.

Haystack's default trusted-module protections stay enabled. The example never passes unsafe=True, expands an allowlist or uses deserialization callbacks. It also refuses a nonempty HAYSTACK_DESERIALIZATION_ALLOWLIST environment variable. The allowlist is an additional protection, not a sandbox.

The SDK buffers the download before the size check. Keep the remote file small and trusted; the 16 KiB check does not limit network buffering. Use the same pinned environment in both workers.

For source-reviewed static configuration, Git is a good home. This recipe covers a configuration file produced during a job and handed to its next worker without a source commit or redeploy. For saved index files, see LlamaIndex snapshots. For progress between data batches, see CSV-to-Parquet batches.

Clean up and billing

uv run python handoff.py cleanup --filesystem "$FS_ID" --run "$RUN" --sha256 "$SHA256"

Cleanup verifies the same digest and known configuration, deletes the YAML using its current ETag and removes the directory only if empty. A wrong hash preserves the file. If another writer changed it, inspect it rather than bypassing the checks. Coordinate deletion with readers.

Inspect and revoke only this recipe's sessions through your backend or the console. Closing the SDK closes HTTP connections, not the server sessions; otherwise they expire normally.

Storage and downloads follow usage pricing. Uploads and requests have no separate tariff. One 977-byte read costs about EUR 0.0000000155, and a full month of those YAML bytes costs about EUR 0.0000001547, plus EUR 0.425 per million entries-month, before tax. These are tariff estimates. They are not infrastructure-cost or margin measurements. Deleting files does not reverse usage; retained bytes remain billed until garbage collection. This recipe creates no account, filesystem or credits.

The companion README has offline test commands and error checks. Set a spending cap for the filesystem before running your own experiments.