Conformal quantile convention probes: harnesses for auditing finite-sample validity in conformal-prediction implementations
Authors/Creators
Description
Measurement harnesses used to audit how Python conformal-prediction implementations resolve the conformal quantile at finite sample sizes.
Distribution-free intervals at finite sample size are indexed by order statistics, and the index alone fixes the coverage attached to it, given the calibration-set size. Numerical quantile functions are not indexed that way: they take a probability level, and every sample-quantile convention turns that level into a position by its own rule. These harnesses measure, per implementation, which index the level actually reaches and the coverage carried by it.
Each script re-derives its own formulas with fractions.Fraction the moment it loads, and a failed check aborts the run. Several of them caught mistakes in hand-derived assertions written by the same person who wrote the check, which is why they are there. Two findings were retracted after such checks contradicted them, and the probes that adjudicated them are included rather than removed.
Simulation only, no third-party library in the loop.
rank_map.py— the nine Hyndman & Fan sample-quantile definitions plus NumPy's four discontinuous aliases, examined as carriers of a coverage guarantee rather than as estimators: delivered rank, the level that recovers the required rank, and the delivered minimum calibration size beside the theoretical one.branch_d_check.py— an uncorrected level versus the required order statistic on identical draws, 200k draws per cell,fractions.Fractionoracle.convention_probe.py— structural branch identification, coverage sweeps with paired standard errors, threshold extraction.fractional_rank.py— delivered coverage tracked against the virtual index the quantile definition computes, rather than against the branch or the (n+1)/n correction.
Closed forms, with a numerical check of each against sampling.
tail_law.py,gpd_tail_law.py— closed forms for the coverage deficit in both extreme-value domains, and their unification: the deficit depends on the tail only through the shape index.randomized_bound.py— the valid randomised bound that exists below the deterministic feasibility floor.horizon_feasibility.py— the floor scaled by forecast horizon, and a residual-alignment off-by-one in a multi-horizon helper, verified three independent ways.conditional_severity.py— group-conditional coverage at a fixed calibration budget.
Installed packages, driven through their public entry points.
run_sktime_river.py— sktimeConformalIntervalsthroughpredict_intervaland riverRegressionJackknifethroughpredict_one, with an oracle independent of both.run_darts.py,run_darts_tighten.py,darts_scoring_path.py— exact rank arithmetic, paired Monte Carlo, and end-to-end runs through a realConformalNaiveModel.verify_statsforecast_rebuttal.py— window mapping forConformalSeasonalPool, and the probe that adjudicated and retracted a finding of the author's own.
Archive data, two arms per cell. Two Monash forecasting collections and two OpenML tabular suites. In every arm the library threshold and a required-rank threshold are built from the same scores and the same point prediction, so the difference isolates the convention.
run_real_data.py,run_real_data_darts.py,run_real_data_statsforecast.py,run_real_data_tabular.py,export_series.py,paired_report.py.
Source inspection, plus a classifier for arbitrary helpers.
helper_census.py— the level-to-rank resolution sites across the audited packages, classified by extracting the threshold each one returns, not by reading it, and verified on disk.conformance_suite.py,certificate.py,certificate_suite.py— a suite that classifies any quantile helper, reporting its branch, the index it reaches, the coverage that index carries, and the least calibration size that honours the level as asked; plus a minimal certificate showing that which calibration sizes a suite must probe can be computed instead of guessed.cross_library.py— the cross-language comparison executed rather than transferred arithmetically; the ecosystem defaults disagree.mapie_clip_reachability.py— whether a clip on the corrected level is reachable through the public API and what that path delivers. The whole infeasible window in exact arithmetic, the first feasible size kept as a control that must read zero, and the result confirmed end to end. It withdraws a retraction of this audit's own: the earlier scan held one keyword fixed and never varied it.lac_crossval_dead_value.py— whether a value a library exposes as a fitted attribute decides anything on the path that computes it. Scaling it by 1000 leaves every returned set identical; a sibling class running identical arithmetic in the identical branch does consult it, and is the control that gives the negative result meaning.conformance_suite.py --oos,oos_conformal_tights.py— the same classifier pointed at four packages the census does not cover, so the instrument meets code it was not written against: one library that computes the required rank and hands it back as a level, one that resolves the requested coverage with no correction, two branches of one helper landing an order statistic apart, and one whose coherence clip decides whether the resolved position reaches the value returned.oos_frame.py— the two populations behind the rows above. On the census side, the thirteen distributions that were downloaded and opened: ten audited, two turned away by the criterion, and one turned away on a reading that running it overturned. On the out-of-census side, the population those four came out of. One verdict per package on the unaudited side of the download denominator, each tied to a path and line number inside a published source distribution, re-opened here where this repository keeps a copy. Twelve read, four carrying a site, two where a fresh pair of eyes broke the earlier reading, six with none. No site count is printed: ruling on how many distinct expressions a package exposes is work this read did not do.frame_sweep.py— whether the package list could have been assembled by querying the index instead of by naming distributions. Two sweeps that fail in opposite directions: every distribution name matched against a stem pattern, then the metadata behind each hit; and the most-downloaded distributions above a monthly floor, read the same way. Both are scored against the known-positive list, so each one's recall is a number rather than a claim. The download-ranked sweep recovers none of it. The sweep also turned up a distribution implementing the behaviour this work recommends, released while the work was under way, which the probe runs rather than reads.oos_prevalence.py— the cost of the site count that was not made, measured before it is spent. A mechanical enumerator is calibrated against the manifest, which supplies every audited site at a file and a line, and the recall it loses is attributed to named mechanisms rather than to a residual: a comparison where an index was expected, a delegation to a callee, a level renamed locally, a level handed to the ordering call as data. Each loss is anchored at the site that demonstrates it. No rate is printed, because a rate carrying that loss would invite the comparison the loss invalidates.second_coder.py— independent re-classification made possible rather than asserted. The criterion, the classifier and every verdict came from one person, so this ships the half somebody else has to supply. Under a fixed seed it picks sites, emits the surrounding source for each, and lists the permitted branch names; a completed file read back yields raw agreement together with Cohen's kappa. Redaction is verified, not assumed: the emitted file is parsed a second time after writing and the run aborts on any surviving verdict or key term. A synthetic completed file drives the scorer end to end, so it is known good before real input reaches it. Concealment here defeats accident only, since every label is public in the manifest. Nothing has come back, so no agreement statistic exists and none is being held back.w8_falsification.py— whether the mechanism generalises beyond conformal prediction to value-at-risk, tolerance bounds and bootstrap intervals. It is present but not dominant in bootstrap intervals.
Every probe writes a plain-text output file that is committed alongside it, and the files in outputs/ are the provenance record for every number the accompanying manuscript reports. The count is deliberately not given here: it was wrong in earlier releases, which said 32 while the directory held more. Upstream reports for every defect claimed, with filing dates, are in DISCLOSURE.md. The research plan, including what has not been established, is in PLAN.md. Third-party library sources are deliberately not redistributed; probe-requirements.txt pins the versions instead.
Files
shivamlalakiya/conformal-coverage-audit-v1.5.15.zip
Additional details
Related works
- Is supplement to
- Software: https://github.com/shivamlalakiya/conformal-coverage-audit/tree/v1.5.15 (URL)
Software
- Repository URL
- https://github.com/shivamlalakiya/conformal-coverage-audit