Validation Module#

The validation module turns “does my simulation converge to the right answer?” into a single, repeatable call. Given a simulation that declares an analytic reference (“oracle”) via analytic_reference(), it runs the simulation and reports whether the Monte Carlo estimate lands on the known value.

Overview#

A Monte Carlo estimator \(\widehat{\theta}_n\) is trustworthy only insofar as you can check it. When a closed-form answer \(\theta\) exists, that answer is an oracle: a checkable ground truth. This module threads the oracle through the stats engine’s existing target machinery and reports two complementary checks:

within_ci – the oracle lies inside the computed confidence interval.
within_tol – the absolute error is within sigma_tol standard errors:

\[\left| \widehat{\theta}_n - \theta \right| \le k \cdot \mathrm{SE}, \qquad \mathrm{SE} = \frac{s}{\sqrt{n}},\]

with \(k = \texttt{sigma\_tol}\) (default 5). At a fixed seed this check is essentially never flaky, yet a genuine implementation bug lands many standard errors away and fails loudly. That makes it an ideal CI gate.

Usage#

from mcframework import validate_convergence, PiEstimationSimulation

report = validate_convergence(PiEstimationSimulation(), 200_000, seed=0)
print(report.status)      # "pass"
print(report.estimate)    # ≈ 3.14159
print(report.abs_error)   # small multiple of the standard error

If a simulation has no oracle, the report’s status is "no-oracle" and no run is performed — a deliberate signal that the simulation is research, not a verifiable demo:

report = validate_convergence(SomeResearchSim(), 10_000)
assert report.status == "no-oracle"

Adding an oracle to a custom simulation#

Override analytic_reference() to return the known expected value of a single draw, and set reference_source to cite it:

import math
from mcframework import MonteCarloSimulation

class PiSim(MonteCarloSimulation):
    reference_source = "Closed form: 4 * P(X^2 + Y^2 <= 1) = pi"

    def single_simulation(self, _rng=None, **kwargs):
        ...

    def analytic_reference(self, **params):
        return math.pi

Module Reference#

Convergence validation against analytic references (“oracles”).

A Monte Carlo simulation is only as trustworthy as your ability to check it. This module turns that check into a first-class, repeatable operation: given a simulation that declares a known answer via analytic_reference(), it runs the simulation and asserts that the estimate converges to that answer.

The implementation is deliberately thin: it reuses the stats engine’s existing target machinery (bias_to_target(), mse_to_target()) and the confidence interval from ci_mean(). The oracle is simply passed through as the context target.

Two complementary checks are reported:

within_ci – the oracle lies inside the computed confidence interval.
within_tol – the absolute error is within sigma_tol standard errors.

within_tol at a fixed seed is the recommended CI gate: it is statistically principled, essentially never flaky, yet a genuine implementation bug lands many standard errors away and fails loudly.

Example#

>>> import math
>>> from mcframework.validation import validate_convergence
>>> # report = validate_convergence(PiSim(), 200_000)
>>> # report.status, round(report.estimate, 2)

Classes#

ConvergenceReport

Outcome of a convergence check against a simulation's analytic reference.

Functions#

validate_convergence

Check that a simulation's Monte Carlo estimate converges to its analytic reference.