τ-bench Validation Story

Validating tau bench benchmark setup

Before comparing Talkdesk, we checked that our setup reproduces a published τ-bench result.

Milestone 1: validate the benchmark setup with the adapter

Verdict

Milestone 1 complete: setup validated. pass¹ matches the published result. pass⁴ is lower, but consistent across our Azure runs, and the adapter scores in the same range as native τ-bench. We are ready to run GPT-4.1-mini and Talkdesk.

Run settings (same for every run)

Sources: Saved run configuration · Published run configuration

AgentGPT-5.2, high reasoning

Simulated customerGPT-5.2, low reasoning

Tasks50 Airline tasks × 4 tries = 200 conversations

ProviderAzure OpenAI (published run: OpenAI)

SeedsSame fixed seeds in every run

Parallel3 conversations at a time (our runs)

What we did

Sources: Adapter implementation · Offline tests

1Pick a referenceNo published GPT-4.1-mini result, so we used the published GPT-5.2 resultDone

2Run native τ-benchOn Azure OpenAI, twice, 200 conversations eachDone

3Build the adapterTools served through our MCP server, like Talkdesk will useDone

4Run the adapterSame tasks, same settings, compared with nativeDone

5GPT-4.1-mini native baselineNative τ-bench on Azure with GPT-4.1-miniDone

6GPT-4.1-mini adapter baselineThe same model through adapter based execution and the MCP serverDone

7Build the Talkdesk adapterConnect Talkdesk to the same MCP serverUpcoming

8Run TalkdeskSame tasks and simulated customer, compared with GPT-4.1-miniUpcoming

Results: pass¹ matches, pass⁴ is lower

pass^k = share of tasks that pass on all k tries, estimated from each task's 4 tries. Every run loses points as k grows; ours follows the same slope as native Azure run 2.

Run	pass¹	pass²	pass³	pass⁴
Published (OpenAI)	83%	78.33%	75%	72%
Native Azure, run 1	83%	75.67%	69.5%	64%
Native Azure, run 2	80%	70.67%	63.5%	58%
Adapter based	78%	67.67%	61%	56%

Results:Published Native 1 Native 2 Adapter

pass¹ matchesNative Azure scored 83% and 80% against the published 83%. Adapter based scored 78%, in the same range.

pass⁴ is lowerEven native τ-bench on Azure scored 64% and 58% against the published 72%. Adapter based (56%) is in the same range as those Azure runs.

pass² and pass³ follow the same patternEach extra required try lowers every run. Adapter based tracks native Azure run 2 within 3 points at every level.

Same setup, different scoresThe two native runs had identical settings, yet differ by 3 points on pass¹ and 6 on pass⁴.

Observed score range

Sources: Calculated score differences

The adapter scored 78% pass¹ and 56% pass⁴, within 3.5 and 5 percentage points, respectively, of the two native Azure runs’ average.

How each task scored, out of 4 tries

Run	4/4	3/4	2/4	1/4	0/4	Passed conversations
Published (OpenAI)	36	6	1	2	5	166 / 200
Native Azure, run 1	32	11	2	1	4	166 / 200
Native Azure, run 2	29	11	5	1	4	160 / 200
Adapter based	28	10	5	4	3	156 / 200

Sources:Task scores per run (CSV)Run distributions

4 of 43 of 42 of 41 of 40 of 4

Same pass¹, different pass⁴Published and native run 1 both passed 166 of 200 conversations, so pass¹ is identical (83%). Native run 1 spread its failures across more tasks: 11 tasks at 3 of 4 instead of 6. That alone takes pass⁴ from 72% down to 64%.

The numbers behind the scores

Sources:Calculations and paired comparisons Calculation methodology

1. How pass¹ to pass⁴ are calculated

For each task we ask: if we picked k of its 4 tries at random, how likely is it that all k passed? The run's score is the average of that across the 50 tasks.

A task that passed…	pass¹	pass²	pass³	pass⁴
4 of 4 tries	100%	100%	100%	100%
3 of 4 tries	75%	50% (3 of 6 pairs)	25% (1 of 4 triples)	0%
2 of 4 tries	50%	17% (1 of 6 pairs)	0%	0%
1 of 4 tries	25%	0%	0%	0%
0 of 4 tries	0%	0%	0%	0%

Worked example, adapter based pass²: (28 tasks × 100% + 10 × 50% + 5 × 17% + 4 × 0% + 3 × 0%) ÷ 50 = 67.67%

2. Gap to native Azure

	pass¹	pass⁴
Native Azure average (runs 1 and 2)	81.5%	61%
Adapter based	78%	56%
Gap	−3.5 points	−5 points

3. Is the gap meaningful?

Compare the adapter's gap with the gap between two native runs where nothing was changed.

83%Native run 1 (pass¹)

80%Native run 2, same setup, 3 points lower by chance alone

78%Adapter based, 2 points below native run 2

The adapter's gap is about the same size as the difference between two identical native runs, so it can't be blamed on the adapter. pass⁴ shows the same thing: native runs 64% and 58% (6 points apart), adapter based 56% (2 points below run 2).

4. How much identical runs vary

32 / 200conversations flipped between pass and fail across the two native runs (16%)

8 lost, 5 gainedperfect tasks between native run 1 and run 2

0settings changed between the two runs

How the adapter differs from native

Native τ-bench

Customer→GPT agent→Tools inside τ-bench

Adapter based

Customer→GPT agent→Our MCP server→τ-bench scores

Chat Completions APIThe same API as native. The only difference left is where the tools run.

What we checked on the adapter run (validation summary)

1,984

tool calls through MCP, all with the same result as native

200 / 200

conversations completed on the adapter

scoring differences when every conversation was re-scored

Why pass⁴ moves more than pass¹

Sources:Customer-stop review Failed conversation endings

pass⁴ is all or nothing. A task counts only if all 4 tries pass.

4 of 4 → counts for pass⁴

3 of 4 → task lost for pass⁴

One slip costs 2 points on pass⁴ but only 0.5 on pass¹. About 20 of the 50 tasks pass on some tries and fail on others.

One common slip comes from τ-bench's simulated customer:

AgentPlease reply YES to confirm you want me to cancel reservation Z7GOZK now.

CustomerYES. ###STOP###

Chat ends. Nothing is cancelled, so the task fails.

This happened 4 times in native run 1, 7 times in native run 2 and 7 times on the adapter, so it affects both sides.

Re-run test: we replayed only those conversations and gave the agent one more turn after the customer's "YES".

3 / 4Native run 1 completed

7 / 7Native run 2 completed

7 / 7Adapter based completed

17 of 18 finished the task correctly. This was a diagnostic only; the official scores above are unchanged.

Diagnostic results:Adapter first case Adapter remaining cases Native 1 Native 2

Three tasks almost never pass

Sources: Per-task results across runs · Published task results

0 / 20Task 7tries passed across all five runs

0 / 20Task 44tries passed across all five runs

2 / 20Task 32tries passed across all five runs

These tasks are hard for GPT-5.2 itself, in the published run, native Azure and adapter based alike. They lower every run's score by the same amount, so they don't affect the comparison.

Cost and time per full run

Run	Agent	Customer	Total	Time
Published (OpenAI)	$22.76	$3.20	$25.95	n/a
Native Azure, run 1	$20.05	$2.97	$23.02	4.7 h
Native Azure, run 2	$19.45	$2.85	$22.30	2.1 h
Adapter based	$18.13	$3.00	$21.13	1.9 h

Costs are token-based estimates, not invoices.

Sources:Local run cost and timing analysis Published usage and costs

Sources

Every number on this page comes from these saved files in the project repository (tau2-bench/). Click a source name to open the file. The fingerprint is the start of each file's SHA-256 hash; if the file changes, the fingerprint changes.

Result	Source	Fingerprint
Published GPT-5.2 scores (83% / 72%)Saved published trajectories and leaderboard entry	Published conversations Published leaderboard entry	308a133d880def13
Native Azure, run 1 (83% / 64%)Full results, 200 conversations	Native run 1 results	436e45285cccd736
Native Azure, run 2 (80% / 58%)Full results, 200 conversations	Native run 2 results	bb45befbc34d2466
Adapter (78% / 56%)Full results, 200 conversations	Adapter results	e2786602319924c5
1,984 tool calls matched, 0 re-scoring differences, 200 sessionsValidation summary written after the adapter run	Adapter validation summary	87b2853820e7fbf1
Task breakdown, pass² and pass³, gaps, 32/200 flipsCalculation methodology and results	Statistical analysis Task scores per run Calculation methodology	ae88582039867a1c
Customer says YES and ends the chat (4 / 7 / 7 cases)Manual review of all 118 failed conversations	Customer-stop review Failed conversation endings	—
Re-run test, 17 of 18 completedContinued conversations, each re-scored	Adapter continuation: first case Adapter continuation: remaining cases Native run 1 continuation Native run 2 continuation	—
Adapter code and testsInitial suite: 72 tests; expanded suite: 80 tests	Adapter implementation Offline tests Implementation commit: 9395f8c	—

How to check: run sha256sum <file> in the repository and compare the first 16 characters. Every run's scores were also re-calculated offline from these files with τ-bench's own scorer, with no differences.

Milestone 1 progress

1GPT-4.1-mini baselines · DoneNative and MCP results in the GPT-4.1-mini Adapter Validation tab

2TalkdeskSame MCP server, tasks and simulated customer

3Compare on pass¹ through pass⁴Evaluate performance across all four pass metrics

GPT-4.1-mini: native versus MCP adapter

The v0.1.3 adapter scored 46% on pass¹ and 26% on pass⁴, versus native’s 45% and 20%. The earlier local native and MCP runs are retained below for comparison.

Current status

Four local GPT-4.1-mini runs complete. The v0.1.3 native and MCP adapter results are included alongside the earlier runs and the v0.1.3 repository reference.

Original local run configuration

AgentAzure GPT-4.1-mini

Simulated customerAzure GPT-4.1

Model argumentsTemperature 0 for both

Evaluation set50 airline base tasks × 4 trials

Seed / concurrency300 / 3, no workers

Limits200 steps / 10 tool errors

The v0.1.3 repository reference records GPT-4.1-mini-2025-04-14 and customer GPT-4.1-2025-04-14. Our saved Azure deployment names do not independently verify deployed model versions. Matching run settings does not establish identical provider or repository behavior.

Performance across all four pass metrics

Repository reference · OpenAI · v0.1.3Native Azure · run 1Native baseline · v0.1.3MCP adapter · Chat CompletionsMCP adapter · Chat Completions · v0.1.3

Run	Successes	pass¹	pass²	pass³	pass⁴	4/4 tasks	3/4 tasks	2/4 tasks	1/4 tasks
Repository reference · OpenAI · v0.1.3	101/200	50.5%	39.33%	32%	26%	13/50	12/50	4/50	5/50
Native Azure · run 1	89/200	44.5%	32%	27%	24%	12/50	6/50	6/50	11/50
Native baseline · v0.1.3	90/200	45%	30%	23.5%	20%	10/50	7/50	9/50	11/50
MCP adapter · Chat Completions	87/200	43.5%	29%	22.5%	20%	10/50	5/50	12/50	8/50
MCP adapter · Chat Completions · v0.1.3	92/200	46%	35%	29.5%	26%	13/50	7/50	6/50	7/50

Each task is run four times. 1/4 means exactly one success, 2/4 two, 3/4 three, and 4/4 all four. A count such as 12/50 means 12 of the 50 tasks belong to that group. pass¹ measures average single-trial success; pass⁴ is the percentage of tasks passing all four trials. passᵏ averages C(successful trials, k) / C(4, k) across 50 tasks.

Original MCP run: execution checks

200 / 200

Scored conversations with distinct MCP sessions

1,455

Recorded tool calls in completed trajectories

Logged result mismatch or infrastructure/accounting errors

All 200 conversations ended with customer STOP; all task/trial pairs are present once. The 42 tool errors were ordinary domain errors, including insufficient payment balance and invalid booking inputs. Earlier connection-refused attempts are absent from the final scored trajectories. Checks cover saved results and their corresponding 200 session logs; independent strict re-scoring has not been performed for this run.

Original local runs: timing and estimated cost

Run	Successes	4/4 tasks	Duration	Agent	Customer	Total
Native Azure	89/200	12/50	39m 17s	$1.83	$2.58	$4.41
MCP adapter Azure	87/200	10/50	43m 15s	$1.90	$2.56	$4.46

Costs are saved model estimates, not invoices. Duration spans the earliest conversation start through the latest end.

Sources

The v0.1.3 results follow the confirmed complete-run comparison table.

Download the original results used to calculate these tables. Fingerprints are the first 16 characters of each source’s SHA-256 hash.

Source	Fingerprint
Repository reference (OpenAI · v0.1.3) results (JSON)	41788956547a100f
Native Azure results (JSON)	50b2da4cfde27447
MCP adapter Azure results (JSON)	a7af45017aa3390a

Next: Talkdesk evaluation

1 · DoneNative and MCP baselinesMatched Azure model settings and customer

2 · PendingTalkdesk configurationConfigure airline policy, model and MCP skills; validate one conversation

3 · PendingCompare all four metricsSame 50 tasks, four trials and GPT-4.1 customer

τ-bench · Airline domain · Milestone 1

Run settings (same for every run)

What we did

Results: pass¹ matches, pass⁴ is lower

Observed score range

How each task scored, out of 4 tries

The numbers behind the scores

1. How pass¹ to pass⁴ are calculated

2. Gap to native Azure

3. Is the gap meaningful?

4. How much identical runs vary

How the adapter differs from native

Native τ-bench

Adapter based

What we checked on the adapter run (validation summary)

Why pass⁴ moves more than pass¹

Three tasks almost never pass

Cost and time per full run

Sources

Milestone 1 progress

Original local run configuration

Performance across all four pass metrics

Original MCP run: execution checks

Original local runs: timing and estimated cost

Sources

Next: Talkdesk evaluation