AIRLINE BENCHMARK · IMPLEMENTATION OVERVIEW
How the MCP adapter works
The adapter connects the native τ-bench agent to our MCP tool server. It preserves native model request construction and adds MCP routing, result comparison and conversation isolation.
Conversation flow
- The agent calls Azure GPT‑5.2 through Chat Completions with the native prompt, conversation history and tool definitions.
- When the model requests a tool, the adapter sends the call to the MCP server using that conversation’s isolated session.
- τ-bench also executes the requested tool locally. This local copy provides the database state used for scoring.
- The adapter compares the local result with the MCP result. The model receives the MCP result for its next turn.
- At conversation completion, the adapter checks tool-call accounting and closes the session.
Controls implemented
| Control | Purpose |
|---|---|
| Fail-closed execution | An MCP infrastructure failure raises an error instead of silently substituting a local result. |
| Result comparison | Differences between MCP and local tool results are recorded. |
| Separate sessions | Each conversation maintains its own tool state, supporting three concurrent conversations. |
| Native scoring | The benchmark evaluates the locally executed database state and saved conversation. |
Observed full-run outcome
The current adapter completed 200 conversations in distinct sessions. All 1,984 tool calls matched local results, with zero MCP infrastructure errors. Two ordinary domain tool errors were recorded. Strict offline re-scoring produced zero differences.
These checks validate tool execution and scoring for the tested airline configuration. They do not establish statistical equivalence of agent success rates or reproduction of the published GPT‑5.2 score.
Implementation scope
The adapter uses native Chat Completions model calls. The implementation was recorded in commit 9395f8c. Core τ-bench source under src/tau2 was not modified for this integration. This page is a readable summary of the implementation, rather than source code.