AIRLINE BENCHMARK · CALCULATION METHODOLOGY

How scores and differences are calculated

pass¹ through pass⁴

Each run contains 50 tasks with four trials per task. For a task with s successful trials, its passᵏ contribution is the number of successful groups of k trials divided by the number of possible groups of k trials: C(s,k) / C(4,k). Contributions are averaged across all 50 tasks.

For example, a task that passed three of four trials contributes 75% to pass¹, 50% to pass², 25% to pass³ and 0% to pass⁴. A task that passed two trials contributes exactly 1/6 to pass².

Native comparison

The two native runs averaged 81.5% pass¹ and 61% pass⁴. The current MCP run scored 78% and 56%, giving differences of −3.5 and −5 percentage points.

Uncertainty analysis

The saved analysis used a paired task percentile bootstrap with 100,000 resamples and seed 20261012. Each resample retained all four trials and all compared runs for a selected task together.

These intervals describe task sampling under stable-run assumptions. They do not provide prediction bounds for future runs on the fixed benchmark, isolate provider effects or establish statistical equivalence.

Continuation diagnostics

Diagnostic recoveries were analysed separately. Substituting those recoveries into original totals is an explanatory calculation, not a new full benchmark or a replacement official score.