mountable.io

Cookbook: keep a DSPy evaluation after the worker exits

Save a tiny DSPy evaluation's inputs, predictions and results as JSON files. A fresh reader checks their hashes and recomputes the same score.

A percentage in a log does not show which evaluation cases matched. If the worker that ran the evaluation exits, its local dataset and predictions may be gone. Save those files before it exits so another process can inspect them.

mountable.io is a persistent filesystem for AI agents. This recipe saves a small evaluation bundle there, then checks it from a fresh read-only process. mountable.io maintains the runnable example separately from DSPy.

Prerequisites

Use Python 3.11+ and uv, an existing mountable.io account with a saved payment method, an existing filesystem and an API key that can access it. Set MOUNTABLE_API_KEY in your private environment or secret manager. Never put the key in command arguments or Git.

Storage and downloads follow the normal usage prices. This recipe creates no account, filesystem, credits or model subscription.

git clone https://github.com/writeitai/mountable.git
cd mountable/examples/dspy-evaluation
uv sync --locked
export MOUNTABLE_FILESYSTEM_ID="your-existing-filesystem"
RUN=$(python3 -c 'import secrets; print(secrets.token_hex(8))')

DSPy 3.4.1 and mountable-sdk 0.3.0 are pinned with a committed lockfile. The fixture uses a dspy.Module that returns fixed dspy.Prediction answers, three real dspy.Example inputs and an exact-match metric through dspy.Evaluate. It makes no model calls. Two answers match, so the score is 66.67%. This is a storage example, not a model benchmark.

The filesystem stays in backend configuration. The validated run id selects only dspy-evaluations/<run>/. In an application, authenticate the real user before choosing that user's filesystem and run id. For separate filesystems and hard spend caps per user, see the capped workspace guide.

Save the evaluation files

uv run python evaluation.py write --run "$RUN"

The writer evaluates the three synthetic cases and uploads:

  • dataset.json: inputs and expected answers.
  • results.json: predictions, per-case exact matches and the aggregate score.
  • manifest.json: format, DSPy version, run id, byte lengths and SHA-256 hashes.

The entire bundle is bounded to 1 MiB before upload. The fixture is about 1 KiB. All writes use If-None-Match: *, so repeating an id leaves the first files in place. The manifest is uploaded last. Only after every upload succeeds does the writer print manifest_sha256=<64 lowercase hex characters>. DSPy may also log its average metric to stderr.

If an upload fails, inspect the run before retrying. Partial data files can remain, but a data-upload failure does not publish a manifest. Cleanup can remove those known paths. Use a new run id for the next attempt.

Read from a fresh process

Pass the run id and the writer's manifest digest through a trusted backend channel. Set SHA to the digest the writer printed, then start a new process:

uv run python evaluation.py read --run "$RUN" --manifest-sha256 "$SHA"

Expected output:

match cases=3 score_percent=66.67

The reader opens a read-only SDK handle. It first checks the manifest's digest against the value from the writer. Then it checks fixed file names, format, versions, sizes, hashes and the exact fixture schemas. It refuses changed or missing files. It recomputes each match from the saved expected answer and prediction, then checks the aggregate rounded to two decimal places.

The reader does not call the predictor or Evaluate again. Hashes check byte identity when the digest comes from a trusted source. The score check verifies arithmetic, not the truth of arbitrary model answers. JSON is parsed as data; downloaded code is never executed or unpickled. The files are not DSPy program serialization, an optimizer checkpoint or training state.

Check permissions and remove the bundle

A second writer must fail with precondition_failed, leaving the first bundle readable. A read-only write must fail with read_only:

uv run python evaluation.py write --run "$RUN"
uv run python evaluation.py ro-write --run "$RUN"

Read the bundle again to confirm it still matches. Then clean up this run:

uv run python evaluation.py cleanup --run "$RUN"

Cleanup uses only fixed paths and ETag-conditional file deletes. It also handles partial uploads. Logical deletion can leave retained bytes billed until reclamation. SDK token sessions normally expire after an hour; backend operators can revoke their own job sessions through the management API.

The normal prices apply, with existing caps and quotas in force. A 402 filesystem_spending_cap_reached stops that filesystem's billable work. There is no model-provider charge for the fixture.

Offline checks and sources

uv run pytest -q

The tests use real DSPy and the released SDK over a fake Files API on an ephemeral loopback port. They start separate writer and reader processes, check read-only and create-only behavior, refuse missing or changed files and invalid schemas, and fail each upload in turn. They make no cloud calls.

APIs checked on 2026-10-27 against the installed DSPy 3.4.1 release and its Evaluate source, the modules guide and the Evaluate documentation.