Keep the Files Behind Your DSPy Evaluation Score

Save a DSPy evaluation's inputs, predictions and per-case results before the worker exits. A fresh reader checks the same files and recomputes the score.

You run a DSPy evaluation in a short-lived worker. It finishes, prints a score, and exits. The inputs and predictions were saved on that worker's disk.

The next reviewer needs those records to understand which cases matched. A percentage in a log is not enough, and running the evaluation again produces a new set of results to review.

mountable.io is a persistent filesystem for AI agents. Save the finished evaluation files there so the writer and a later reader use the same filesystem, without copying files by hand or adding a separate results store. mountable.io maintains this example separately from DSPy.

Three cases with answers you can inspect

The example uses three synthetic arithmetic cases. A fixed dspy.Module returns a stored answer for each question, and dspy.Evaluate compares those predictions with the expected answers in dspy.Example records. It makes no model calls.

Two answers match and one is deliberately wrong. The fixture's score is 66.67%. That number demonstrates the file handoff: it is the same before and after the worker exits.

Upload the records before announcing completion

The writer saves the inputs and results as JSON. It uploads a manifest with their sizes and hashes only after both data files are stored.

from mountable_sdk import Mountable
 
with Mountable() as client:
    fs = client.filesystem(filesystem_id, "rw")
    fs.write(f"{run_path}/dataset.json", dataset, if_none_match=True)
    fs.write(f"{run_path}/results.json", results, if_none_match=True)
    fs.write(f"{run_path}/manifest.json", manifest, if_none_match=True)

Each file is created only once. A second create leaves the first bytes in place. If a data upload fails, there is no manifest to announce a completed bundle to the reader.

Check the same score from another process

Give the next process the run id and the file hash the writer printed. It opens a read-only handle, checks the files, and recomputes the exact matches from the saved records. The result is:

match cases=3 score_percent=66.67

The reader can now inspect the individual predictions behind that percentage. The full guide shows how to check the bundle and remove it when the review ends.

Small files, ordinary usage prices

The fixture stores about a kilobyte of synthetic data. Storage is $0.20 per GiB-month at the normal usage prices, and downloads are billed too. Delete the files when you no longer need them; retained bytes can remain billed until reclamation. The fixture calls no model provider.

• Follow the DSPy guide: write, read from a new process and clean up.
• Run the example: the evaluator, reader and offline tests.