# Airline customer-simulator termination review 12 October 2026. Offline review only; no paid calls, benchmark edits or score changes. ## Confirmed finding The customer sometimes approves an action and ends the conversation in the same message, before the agent can execute that action. This occurs in native runs and in the Chat Completions MCP run. All 118 failed-conversation endings were manually read: 34 native v1, 40 native v2 and 44 MCP. The review checked the preceding agent message, terminal customer message, write calls already made, and recorded reward breakdown. The 50 saved task definitions were also verified identical across these three runs. | Run | Failed conversations reviewed | Approval + STOP with write still pending | Affected tasks | |---|---:|---:|---:| | Native v1 | 34 | 4 | 4 | | Native v2 | 40 | 7 | 6 | | Chat Completions MCP | 44 | 7 | 6 | Each run has 200 conversations. These are confirmed cases among failures, not an exhaustive classification of successful conversations or an estimate of all possible simulator mistakes. The equality of the native v2 and MCP counts does not establish statistically equal simulator-error rates. ## Concrete example MCP task 19, human trial 2 (zero-based trial 1): - Agent: “Please reply YES to confirm you want me to cancel reservation Z7GOZK now.” - Customer: “YES. ###STOP###” - No cancellation write was made. The DB reward is 0. Native v2 task 20, human trial 2 has the same pattern for a new booking. The agent lists the booking and asks for YES; the customer replies “YES.###STOP###”. No booking write follows. This timing is confirmed by source: `UserSimulator.is_stop()` recognizes STOP, TRANSFER and OUT-OF-SCOPE anywhere in customer content. `Orchestrator.step()` marks the simulation done immediately when the customer emits such a message. It records that customer message but never runs the next agent turn. The customer's saved simulation guidelines say STOP should be generated when the instruction goal is satisfied. Approval of an unexecuted required action does not satisfy that goal. These are observed premature customer terminations; they are not evidence of an MCP execution failure. ## Exact cases Trial numbers below are human-readable (1–4). - Native v1: task 23/trial 3; 24/1; 29/3; 42/3. - Native v2: task 17/trial 4; 20/2; 23/4; 24/2; 24/4; 25/1; 33/3. - MCP: task 14/trial 2; 19/2; 20/3; 23/3; 23/4; 24/2; 30/2. Seven MCP cases have no previous write calls. In native v1 all four have no previous write calls. In native v2, six have none; task 33 has already changed flights but stops on approval before adding the required baggage. There is one additional clear MCP timing case: task 18/trial 4 completed the required cabin updates (DB reward 1), then the customer asked for the required total savings and included STOP. The agent was unable to answer; COMMUNICATE reward is 0. This is separate from approval of a pending database write. Three other endings combine a requested/approved write with TRANSFER rather than STOP: native v1 task 9/trial 3; native v2 task 37/trial 3 and task 44/trial 4. They are kept separate because transfer context and prior agent/customer errors can complicate attribution. Task 44 also specifies an inappropriate payment method. They are not included in the 4/7/7 primary counts. ## What this establishes The benchmark measures the combined stochastic agent/customer interaction. Some recorded failures include a customer that prematurely removes the agent's chance to act. This weakens an interpretation that every failed conversation reflects an agent or MCP defect and supports investigating repeatability before assigning a score gap to MCP. It does not establish that the agent would have executed correctly if given another turn. Earlier agent errors or customer instruction drift can coexist. No hypothetical success is awarded and no adjusted scores are calculated. This timing review is not a complete causal audit of all 118 failures. The counts do not explain the entire native–MCP score gap: native also suffers the same failure pattern. Other failure endings include completed but incorrect workflows, policy transfers, refusal/DOB issues, and customer information drift. Those require separate task and policy analysis before assigning final causes. ## Next decision Keep the current results and scores intact. A third unchanged native run can measure another observed repetition, but it will not remove the termination problem. Decide whether the goal is reproducing official τ-bench as implemented or separately diagnosing/improving its simulator. If simulator instructions or termination behavior are changed, label the resulting experiment as a changed benchmark and rerun both native and MCP under the same new rules. No such changes or paid runs have been made. ## Evidence and reproduction - [All 118 annotated failure endings](evidence/airline-mcp-statistical-analysis/failure_endings_full.json) - [Counts, definitions and simulation IDs](evidence/airline-mcp-statistical-analysis/termination_review_summary.json) - [Offline extraction and saved manual annotations](evidence/airline-mcp-statistical-analysis/review_termination.py) Run the extraction and assertion checks with the existing WSL environment: ```bash .venv/bin/python platform-comparission-docs/evidence/airline-mcp-statistical-analysis/review_termination.py ``` Saved input SHA256 values are preserved per record. The script checks 118 unique failure IDs, identical task definitions, final user messages, STOP presence and failed DB rewards in the main category. The annotations came from manual review; rerunning the extraction does not perform an independent semantic review.