Benchmark Module#

The benchmark module turns “how fast is each backend on my simulation?” into a single, repeatable, plottable operation. Given a simulation, it times the same workload across every available execution backend (sequential, thread, process, Torch CPU, and Apple-Silicon MPS) and returns a structured, JSON-serializable report.

Overview#

Backends are described by BackendSpec, so the benchmark matrix is data, not code. default_backends() builds the device-aware default set (Torch / MPS / CUDA backends are added only when their device is actually available), and run_suite() sweeps the sizes × specs grid. The measurement core depends only on NumPy; plot_benchmarks() imports matplotlib lazily, so plotting stays an optional extra (pip install mcframework[viz]).

Adding a new accelerator later — NVIDIA CUDA, for instance — is a single extra BackendSpec plus a device gate in default_backends(); no new benchmark script is required.

Usage#

from mcframework import PiEstimationSimulation, run_suite, default_backends
from mcframework.benchmark import plot_benchmarks

sim = PiEstimationSimulation()
report = run_suite(sim, [1_000, 10_000, 100_000], default_backends())

print(report.summary_table())     # execution time + speedup table (N/A-aware)
fig = plot_benchmarks(report)      # four-panel performance figure
data = report.to_dict()           # JSON-serializable artifact

The same workflow is available from the command line:

mcframework-benchmark --quick --save backend_benchmark.png
mcframework-benchmark --sizes 1000,10000,100000 --json report.json --no-show

Module Reference#

Backend benchmarking for Monte Carlo simulations.

A simulation framework is only as compelling as the speedups it can demonstrate. This module turns benchmarking into a first-class, repeatable operation: given a simulation, it times the same workload across every available execution backend (sequential, thread, process, Torch CPU, Apple-Silicon MPS) and returns a structured, plottable, JSON-serializable report.

The measurement core is dependency-light (NumPy only). Plotting lives in plot_benchmarks(), which imports matplotlib lazily so the library never carries a hard plotting dependency (install mcframework[viz] to plot).

Design#

Backends are described by BackendSpec, so the matrix is data, not code. Adding NVIDIA CUDA later is a single extra spec plus a device gate in default_backends() – no new benchmark script required.

Example#

>>> from mcframework import PiEstimationSimulation
>>> from mcframework.benchmark import run_suite, default_backends
>>> sim = PiEstimationSimulation()
>>> report = run_suite(sim, [1_000, 10_000], default_backends())
>>> print(report.summary_table())

Classes#

`BackendSpec`	Description of one execution backend to benchmark.
`BenchmarkResult`	Timing outcome for one (backend, size) cell.
`BenchmarkReport`	Collection of `BenchmarkResult` rows plus run context.

Functions#

`system_info`	Capture host + accelerator details for benchmark provenance.
`default_backends`	Build the device-aware default backend matrix.
`benchmark_run`	Time one (backend, size) cell.
`run_suite`	Benchmark `sim` across the `sizes` x `specs` matrix.
`plot_benchmarks`	Render a clean four-panel benchmark figure.
`main`	Command-line entry point for `mcframework-benchmark`.