All completed full runs, side by side
Airline base split · 50 tasks × 4 trials · 200 scored conversations per run.
| Agent model | Run / execution path | Customer model | Successes | pass¹ | pass² | pass³ | pass⁴ | 4/4 tasks | 3/4 tasks | 2/4 tasks | 1/4 tasks |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.2 | Repository reference · OpenAI | gpt-5.2 | 166/200 | 83% | 78.33% | 75% | 72% | 36/50 | 6/50 | 1/50 | 2/50 |
| GPT-5.2 | Native Azure · run 1 | azure/gpt-5.2 | 166/200 | 83% | 75.67% | 69.5% | 64% | 32/50 | 11/50 | 2/50 | 1/50 |
| GPT-5.2 | Native Azure · run 2 | azure/gpt-5.2 | 160/200 | 80% | 70.67% | 63.5% | 58% | 29/50 | 11/50 | 5/50 | 1/50 |
| GPT-5.2 | MCP adapter · Chat Completions | azure/gpt-5.2 | 156/200 | 78% | 67.67% | 61% | 56% | 28/50 | 10/50 | 5/50 | 4/50 |
| GPT-4.1-mini | Repository reference · OpenAI · v0.1.3 | gpt-4.1-2025-04-14 | 101/200 | 50.5% | 39.33% | 32% | 26% | 13/50 | 12/50 | 4/50 | 5/50 |
| GPT-4.1-mini | Native Azure · run 1 | azure/gpt-4.1 | 89/200 | 44.5% | 32% | 27% | 24% | 12/50 | 6/50 | 6/50 | 11/50 |
| GPT-4.1-mini | Native baseline · v0.1.3 | azure/gpt-4.1 | 90/200 | 45% | 30% | 23.5% | 20% | 10/50 | 7/50 | 9/50 | 11/50 |
| GPT-4.1-mini | MCP adapter · Chat Completions | azure/gpt-4.1 | 87/200 | 43.5% | 29% | 22.5% | 20% | 10/50 | 5/50 | 12/50 | 8/50 |
| GPT-4.1-mini | MCP adapter · Chat Completions · v0.1.3 | azure/gpt-4.1 | 92/200 | 46% | 35% | 29.5% | 26% | 13/50 | 7/50 | 6/50 | 7/50 |
Reading the scores: Each task is run four times. 1/4 means it passed exactly once; 2/4 means twice; 3/4 means three times; 4/4 means all four times. A cell such as 12/50 means 12 of the 50 tasks fell into that group. pass¹ is average single-trial success. pass⁴ is the share of tasks succeeding in all four trials. For k = 1–4, passᵏ = average across tasks of C(successful trials, k) / C(4, k).
Configuration: GPT-5.2 uses high agent reasoning and low customer reasoning. GPT-4.1-mini uses GPT-4.1 as customer, with temperature 0 for both. Recorded deployment names do not independently verify underlying Azure model versions.
Validation: Displayed scores recalculated from saved rewards; the scored runs have 200 unique task/trial pairs and the same 50 task IDs.