Oracles and Benchmarks#

A Monte Carlo estimate is only trustworthy if you can check it. mcframework makes “converges to a known answer” a first-class, repeatable operation: when a simulation has a closed-form answer, it declares that answer as an oracle, and validate_convergence() asserts the estimate lands on it.

The idea: oracle = checkable ground truth#

An oracle is a value you already know to be correct: the analytic expectation of a single draw. If the Monte Carlo mean does not converge to the oracle, the estimator is wrong — full stop. This turns correctness into a test instead of a vibe.

mcframework reuses machinery it already has. The oracle is passed straight through to the stats engine as the context target (via run(extra_context={"target": oracle})), which also populates bias_to_target / mse_to_target for free. Validation is a thin pass/fail layer over the confidence interval the engine already computes.

Two complementary checks are reported:

  • within_ci — the oracle lies inside the computed confidence interval.

  • within_tol — the absolute error is within sigma_tol standard errors:

\left| \widehat{\theta}_n - \theta \right| \le k \cdot \mathrm{SE},
\qquad \mathrm{SE} = \frac{s}{\sqrt{n}}, \quad k = \texttt{sigma\_tol}.

within_tol at a fixed seed is the recommended CI gate: it is statistically principled, essentially never flaky (a \(5\sigma\) false failure is a ~1-in-1.7-million event), yet a real implementation bug lands many standard errors away and fails loudly.

Quick start#

from mcframework import validate_convergence, BlackScholesSimulation

report = validate_convergence(BlackScholesSimulation(), 50_000, seed=0, option_type="call")
print(report.status)      # "pass"
print(report.estimate)    # Monte Carlo price
print(report.oracle)      # closed-form Black-Scholes-Merton price
print(report.abs_error)   # within a few standard errors

If a simulation declares no oracle, no run is performed and status is "no-oracle" — a deliberate signal that the simulation is research, not a verifiable demo:

report = validate_convergence(BlackScholesSimulation(), 1_000, exercise_type="american")
assert report.status == "no-oracle"   # American options have no closed form

Built-in oracles#

These ship validated and are pinned by tests/test_oracles.py:

Simulation

Oracle

Reference

PiEstimationSimulation

\(\pi\)

Closed form: \(4\,\Pr(X^2+Y^2\le 1)=\pi\)

PortfolioSimulation (GBM)

\(V_0 e^{\mu T}\)

Risk-neutral GBM expectation

BlackScholesPathSimulation

\(S_0 e^{rT}\)

Risk-neutral GBM terminal price

BlackScholesSimulation (European)

\(C, P\)

Black-Scholes-Merton (1973)

BlackScholesSimulation (American)

none

No closed form → not validated

The Black-Scholes-Merton closed form for a European option is

C = S_0\,\Phi(d_1) - K e^{-rT}\,\Phi(d_2), \qquad
P = K e^{-rT}\,\Phi(-d_2) - S_0\,\Phi(-d_1),

with \(d_1 = \dfrac{\ln(S_0/K) + (r + \tfrac12\sigma^2)T}{\sigma\sqrt T}\) and \(d_2 = d_1 - \sigma\sqrt T\).

Adding an oracle to your own simulation#

Override analytic_reference() to return the known expected value of a single draw under the given parameters, and set reference_source to cite it. Return None when no oracle exists.

import math
from mcframework import MonteCarloSimulation, validate_convergence

class CoinFlipSimulation(MonteCarloSimulation):
    reference_source = "Closed form: E[Bernoulli(p)] = p"

    def __init__(self, p: float = 0.5):
        super().__init__("Coin Flip")
        self.p = p

    def single_simulation(self, _rng=None, **kwargs):
        rng = self._rng(_rng, self.rng)
        return float(rng.random() < self.p)

    def analytic_reference(self, **params):
        return self.p

report = validate_convergence(CoinFlipSimulation(0.3), 100_000, seed=0)
assert report.status == "pass"

analytic_reference receives the same keyword parameters as run / single_simulation, so the oracle can depend on configuration (strike, drift, etc.). Keep the reference a genuine closed form or a cited benchmark — never a second Monte Carlo estimate.

Kinds of reference#

Not every credible reference is a closed form. Set reference_kind to record which flavour of ground truth you are checking against; it is surfaced on every ConvergenceReport:

reference_kind

Meaning

Example

"closed-form"

An exact analytic answer

Black-Scholes-Merton price; \(V_0 e^{\mu T}\)

"benchmark"

A published, cited reference value

A criticality benchmark \(k_\text{eff}\) from a standard suite

"limit"

A known asymptotic / limiting value

A large-\(n\) or zero-variance limit

A "benchmark" reference lets a domain simulation be validated even when no closed form exists: analytic_reference returns the published constant and reference_source carries the citation. The validation machinery is identical — only the provenance of the oracle differs.

Promoting a simulation out of draft#

Rule: a simulation may not leave draft status (be advertised as production-ready, added to the built-in gallery, or recommended in docs) until it has either an analytic oracle or a cited benchmark, wired through analytic_reference and pinned by a convergence test in tests/test_oracles.py.

This is the governance signal the whole system exists to provide. A simulation with no checkable ground truth is research: keep analytic_reference returning None (its reports read no-oracle) and label it as draft until a reference exists.

Benchmark template#

When there is no closed form, validate against a published benchmark in three steps:

  1. Cite the source. Put the full reference in reference_source and set reference_kind = "benchmark".

  2. Reproduce the value. Have analytic_reference return the published constant for the matching parameters (and None for any configuration the benchmark does not cover).

  3. Show convergence. Add a tests/test_oracles.py case asserting validate_convergence(...).within_tol at a fixed seed, and (optionally) add the sim to the convergence gallery.

Worked example: reactor-physics criticality (not implemented here)#

A neutron-transport simulation has no elementary closed form, but the field publishes criticality benchmarks — configurations with an accepted reference \(k_\text{eff}\) (e.g. the ICSBEP handbook, or an analytic infinite-medium \(k_\infty = \nu\Sigma_f / \Sigma_a\)). The pattern would be:

class CriticalityBenchmarkSim(MonteCarloSimulation):
    reference_source = "ICSBEP HEU-MET-FAST-001 (Godiva), k_eff = 1.0000 +/- 0.0010"
    reference_kind = "benchmark"

    def analytic_reference(self, *, config="godiva", **params):
        return 1.0000 if config == "godiva" else None  # only the cited config

    def single_simulation(self, _rng=None, **kwargs):
        ...  # neutron transport estimating k_eff

mcframework ships no neutron-transport implementation: that remains a separate domain effort. What it now provides is the gate — such a simulation is only credible once its k_eff estimate is shown to converge to the cited benchmark.