AI evaluation has never been easier to start. Reproducing it reliably is another story. Developers now have access to more benchmarks, evaluation libraries, model APIs, and agent frameworks than ever before. But keeping the prompt, model, and scoring method fixed doesn’t necessarily make a run reproducible. The execution environment matters too.
Python dependencies change. Local tools drift. Setup steps go undocumented. A workflow that succeeds on one machine may behave differently on another. Most discussions about evaluation focus on what should be measured: benchmarks, scoring methods, or judge models. Much less attention is given to how those evaluations are executed. Yet that execution layer often determines whether someone else can reproduce the same workflow weeks or months later.
When I started exploring Docker Sandboxes, I wasn’t trying to build another evaluation framework. I had a much smaller question.
Could Docker Sandboxes and an SBX Kit make evaluation workflows easier to rerun, inspect, and compare?
That question eventually became the SBX AI Evaluation Kit, an open-source Docker Sandboxes Mixin Kit focused on repeatable execution, structured evaluation records, and runtime evidence. The current implementation does not execute AI models or automatically derive evaluation judgments. Instead, it executes configured commands consistently and preserves evidence of what actually ran.
In Practice
In practice, the workflow starts by choosing where the evaluation command should run through the execution block:
execution:
executor: sbx
command:
- python3
- -c
- print("hello from sbx")
With executor: sbx, the runner delegates command execution to Docker Sandboxes and writes the runtime evidence into the resulting artifact.
The repository is also packaged as an SBX Mixin Kit, so it can be applied when starting a Claude sandbox:
sbx run claude --kit .
The runner reads the configured executor and delegates the command to SBX, which executes it inside the sandbox:
python run_evaluation.py
From Documentation to an Executable Workflow
Each evaluation is defined in a YAML file that describes the evaluation and the command to run. The repository validates that definition, executes it, and produces a structured JSON record of the result. The difference is in what gets recorded. A written evaluation captures what someone intended to do. An execution-backed evaluation captures what actually happened.
Separating Evaluation from Execution
I wanted the evaluation definition to stay independent of where it ran. A workflow written during local development shouldn’t need to change simply because it later executes inside Docker Sandboxes.
To keep those concerns separate, I introduced an executor abstraction. The evaluation describes what should run; the executor determines where it runs.
With the local executor, the configured command runs on the host. With the SBX executor, command execution is delegated to Docker Sandboxes. Switching between the two only requires changing the executor configuration, not rewriting the surrounding evaluation workflow.
Figure 1. Evaluation definitions remain independent of the execution environment. The same workflow can use either the local or SBX executor while producing runtime evidence in the same structure.
Capturing Evidence Instead of Assumptions
For each execution, the runner records enough information to inspect what actually happened:
- the selected executor,
- the command that was executed,
- standard output (
stdout) and standard error (stderr), - the exit code,
- and the execution time.
These details are stored in the evaluation artifact. The repository also generates a digest of the evaluation configuration. This creates a deterministic link between the evaluation configuration and the artifact it produced, without trying to replace full experiment-tracking systems.
{
"executor": "sbx",
"command": ["python3", "-c", "print(\"hello from sbx\")"],
"stdout": "hello from sbx\n",
"stderr": "",
"exit_code": 0,
"duration_ms": 120.0
}
Scaling from One Evaluation to Many
Real-world evaluation rarely consists of one isolated run. Teams compare prompts, validate behavior, measure regressions between releases, and test multiple scenarios. That led to evaluation suites.
Rather than changing how an individual evaluation works, a suite groups multiple evaluation definitions into a single repeatable workflow. Each evaluation still produces its own structured artifact, while the suite also generates an aggregated summary of the overall run.
Reusable SBX Kits Beyond Evaluation
The same pattern isn’t limited to evaluation. An SBX Kit can package more than a development environment; it can also package the setup an engineering workflow depends on. The same model could support regression testing, policy checks, security analysis, code-generation experiments, and other workflows that depend on consistent execution and inspectable results.
Conclusion
The SBX AI Evaluation Kit doesn’t replace evaluation frameworks, benchmarks, or scoring systems. Its job is narrower: execute configured evaluation workflows in a way that is easier to rerun and inspect.
The question I came away with is simple: before comparing benchmark scores or choosing a judge model, can someone else reliably run the same workflow under comparable conditions?
You can explore the code, experiment with custom evaluation YAMLs, and run the workflow yourself in the sbx-ai-eval-kit repository on GitHub.
Resources
- SBX AI Evaluation Kit – Source code, example evaluation definitions, and the implementation described in this article.
- Docker Sandboxes documentation – Official documentation for setting up and running Docker Sandboxes.
- Customizing Docker Sandboxes with Kits – Official documentation for extending Docker Sandboxes with reusable Kits.
