Benchmark source summaries

Native Azure run 1

These are original recorded scores. Costs are estimates, not invoices.

Native Azure run 2

These are original recorded scores. Costs are estimates, not invoices.

Current MCP adapter

These are original recorded scores. Costs are estimates, not invoices.

Published GPT‑5.2 airline reference

The published run used OpenAI GPT‑5.2 with high agent reasoning and low customer reasoning. Its recorded arguments omit explicit temperature, as do our local runs.

Adapter execution and scoring checks

These checks establish execution fidelity for the tested configuration; they do not prove statistical performance equivalence.

Score differences and uncertainty

The analysis resampled 50 whole task rows together across runs, preserving four trials per task. It used 100,000 paired task bootstrap samples with seed 20261012.

Intervals describe task sampling under stable-run assumptions. They do not predict variation in future runs on the fixed benchmark or isolate provider effects. Statistical equivalence remains unproven.

Task outcomes across local runs

Values show successful trials out of four for each task. These are original results, without continuation substitutions.

Customer-stop investigation

All 118 failed conversation endings across the three Chat Completions runs were reviewed for premature customer termination. This was a targeted review, not a complete causal classification of all failures.

Observed example

The agent asked for confirmation to cancel a reservation. The customer replied “YES. ###STOP###”. The orchestrator ended the conversation immediately, so no cancellation followed.

This occurs in both native and MCP. Identifying the pattern alone does not guarantee that a continuation would succeed, nor does it explain every failed conversation.

MCP continuation: first case

1 of 1 selected conversations passed after continuation.

Only the final original STOP token was removed. Prior history and tool state were preserved, and continuation used the remaining budgets. Later customer termination behavior was unchanged.

These are selected diagnostic continuations, not fresh full runs or replacements for the official scores. Across all four diagnostic groups, 17 of 18 cases recovered.

MCP continuation: remaining cases

6 of 6 selected conversations passed after continuation.

Only the final original STOP token was removed. Prior history and tool state were preserved, and continuation used the remaining budgets. Later customer termination behavior was unchanged.

These are selected diagnostic continuations, not fresh full runs or replacements for the official scores. Across all four diagnostic groups, 17 of 18 cases recovered.

Native run 1 continuation

3 of 4 selected conversations passed after continuation.

Only the final original STOP token was removed. Prior history and tool state were preserved, and continuation used the remaining budgets. Later customer termination behavior was unchanged.

These are selected diagnostic continuations, not fresh full runs or replacements for the official scores. Across all four diagnostic groups, 17 of 18 cases recovered.

Native run 2 continuation

7 of 7 selected conversations passed after continuation.

Only the final original STOP token was removed. Prior history and tool state were preserved, and continuation used the remaining budgets. Later customer termination behavior was unchanged.

These are selected diagnostic continuations, not fresh full runs or replacements for the official scores. Across all four diagnostic groups, 17 of 18 cases recovered.