AI providers discontinue models frequently, forcing companies to replace them on short notice, often without enough time to validate whether the replacement model performs as well as the one it's replacing, at the same cost and latency. The risk isn't abstract. On the customer side, it means receiving wrong information about a cancellation, a deadline, or a charge. On the company side, it means a cost per session that rises without warning, an SLA that doesn't hold up at scale, or a model-swap decision made under pressure, without data to support it before it reaches production.
This work presents an evaluation pipeline built on the Tech4AI platform, which tests candidate models under full operational parity with the production environment.
Instead of public benchmarks, which test isolated responses against an AI-simulated user or a scoring key built specifically for the test, the pipeline replays synthetic conversations based on real interactions, selected and approved as a quality reference (Base 0), reproducing the exact operational context of the product: the same memory, the same tools, the same prompt. The only variable under test is the model behind the agent (LLM-swap).
Eight models were evaluated, from the GPT-4.1/5 and Gemini 2.5 families and open-weight models such as Kimi-K2.5 and GPT-OSS-120B, across hundreds of executions covering four service instances.
The approval criterion combines quality, task completion, tool precision, latency, and cost, with the hallucination rate acting as an independent elimination filter. Three models were approved. The most relevant finding is the reason why the other five, despite competitive quality, were rejected.
The AI provider market rarely updates an existing model: it typically creates entirely new model families and discontinues the previous ones. A model in use for two years, with an already-validated cost margin, can have its shutdown date announced with only a few months' notice. As a result, migrating to the substitute recommended by the provider itself doesn't always preserve the same performance, the same cost per session, or the same task completion rate.
The risk also isn't limited to the model in isolation: it involves the entire orchestration of an agent's multiple use cases, such as billing, negotiation, and case filing, plus the precision of tool calls and the preservation of financial margins in contracts already signed with clients.
The goal of the test is to put a candidate model in charge of the same situations the team already handles today, and compare the result against a quality reference.
The difficulty lies in making that comparison fairly, without context bias and without depending on real customer traffic. That's what the pipeline exists for.
Before testing any model, it's necessary to define what counts as "good service." To do this, 106 synthetic sessions were manually reviewed, turn by turn, by a human analyst, against a structured script covering routing, adherence to instructions, and correct tool calls.
Only 20 sessions survived that curation, distributed across four different service instances. These 20 sessions form Base 0: the minimum quality bar that every candidate model must match or exceed, accepting paraphrasing and language variation, as long as they preserve the essential information and the resolution of the interaction.
How the test works, step by step.
Each Base 0 session goes through the same flow:
Visually, this complete flow, including session selection and masking already described above, looks like this:
To maintain safety and allow the same test to be repeated multiple times, the pipeline never triggers a real tool during replay: no charge, cancellation, or registration change actually happens. When the candidate model calls the right tool, at the expected step of the reference interaction, the pipeline returns the same response that tool originally gave in Base 0.
When the model calls a different tool than expected, or out of order relative to the reference interaction, the pipeline returns a simulated failure response, not a success. This tells the model the action didn't work, instead of letting it believe it succeeded, and with that clear failure signal, the pipeline discourages a new attempt at the same call, instead of letting the model repeat the action as if it might work next time. Combined with a limit on repetitions per turn, this rule prevents an incorrect tool call from degenerating into a loop within the same session.
This choice has a known side effect: when the model takes a different path than the baseline, but one that's still valid for resolving the interaction, the response it receives from the tool is still generic, not the real response that alternative path would have generated.
Since the only variable that changes from one test to another is the model behind the agent, any difference in performance, latency, or cost is attributable exclusively to the model swap, and not to differences in context or prompt bias. This tool-execution simulation mechanism is summarized visually below:
At every tool call, the replay engine decides: if it matches the expected Base 0 step, it returns the real recorded response; if it diverges, it returns a simulated failure and the session continues normally from there, with the model simply proceeding knowing that action didn't work.
The first layer uses entirely deterministic business rules, such as response format and presence of required fields.
The second combines two hybrid metrics, TCR (task completion) and TCA (tool precision), which start from a deterministic check and add a layer of contextual judgment when the response diverges from what was expected.
The third is a panel of multiple judges (LLM-as-a-Judge), which evaluates the entire session holistically. These judges have a different architecture from the models being evaluated, which prevents a model from favoring responses that resemble its own style (self-preference bias).
In practice, each judge returns output in this JSON format, never in free text:
{
"score": 0.82,
"verdict": "good",
"detailed_metrics": {
"quality_vs_base0": 3,
"tone_clarity": 4,
"hallucination_type": "none"
}
}The scores from multiple judges are combined using the median for the quality score, but use the most severe classification among them to decide hallucination: a single judge flagging critical hallucination is enough to mark the entire session as such, even if the other judges didn't notice the problem.
All conversations used in the evaluation are synthetic datasets, masked before any calls to the providers' APIs.
Conversational agent evaluation today generally relies on one of three approaches, each solving part of the problem, but not the whole problem:
None of these three approaches, on its own, combines the three things this pipeline requires at once: a curated synthetic reference corpus approved through human review, controlled model swapping under real operational parity, and a verdict that weighs quality, latency, and cost together, with hallucination rate acting as an independent safety filter, not diluted into the average.
Once tested, each model is evaluated on four fronts: cost per session, latency, task-resolution capability, and behavioral quality.
These four fronts combine into a Final Score, on a scale of 1 to 4, with different weights: 50% for quality, 30% for cost, and 20% for latency. The faster and cheaper a model resolves the interaction with quality, the higher the score.
But a high score alone isn't enough. The hallucination rate acts as a separate filter, applied after the Final Score: a model is only approved if, in addition to a Final Score of 75% or higher and a Pass Rate of 70% or higher, it has 0% critical hallucinations (fabricated information with direct impact on the user's decision) and at most 2% general hallucinations (minor inaccuracies). A model can have the best score in the study and still be rejected if it exceeds that limit.
Gemini 2.5 Flash, GPT-4.1 nano, and GPT-5 mini scored above 79% on the Final Score, competitive enough for approval if the criterion were only the weighted average of quality, latency, and cost.
Even so, all three were rejected for exceeding the 2% general Hallucination Rate limit set for the project. The hallucination rate, not response quality, was the eliminating factor in these three cases. GPT-5 mini went further: it recorded the only critical hallucination occurrence among the eight candidates (1.6% of sessions), the strictest elimination requirement in the study.
Without telemetry on contracted cost, the pipeline uses the volume of tokens processed per session as a cost proxy. Converting that consumption into monetary value, using each provider's public list prices, changes the picture: GPT-5.4 mini, which receives the best normalized cost score among the eight candidates, actually has the highest estimated Cost Per Session (around US$0.093), about 5.3 times the cost of GPT-5.4 nano (US$0.017) and 5.8 times that of GPT-4.1 nano (US$0.016, the cheapest in the sample).
The cause isn't the volume of tokens processed. It's the list price per output token for GPT-5.4 mini (US$4.50 per million), far higher than that of GPT-5.4 nano (US$1.25 per million) within the same model family.
Since language models are probabilistic, the same model can produce different results in each independent run. Recalculating the verdict run by run, two models rejected in aggregate (Gemini 2.5 Flash and GPT-5 mini) swung between approval and rejection in a nearly even split, which suggests dependence on the specific sample of sessions, not a structural limitation.
GPT-OSS-120B, by contrast, showed the opposite pattern: it never reached approval in any of the runs, a result consistent with an identified structural cause (the model natively uses the Harmony output format, incompatible with the rest of the pipeline's standard, which caused 87% of sessions to end without a recognizable decision).
In conclusion, the final recommendation is to adopt GPT-5.4 mini, with no operational caveat identified in the data collected. GPT-5.4 nano remains an alternative with lower cost and latency, with a narrower safety margin. Kimi-K2.5 is approved on quality criteria and remains a secondary candidate, conditional on operational validation of the deployment with the provider.
This summary covers the main findings, but the full study details points we didn't explore here: the structure of the prompt used by the judge panel (LLM-as-a-Judge), the consistency of the verdict across runs for all eight models, and the complete list of the study's limitations and the technical criteria used in the final decision matrix.
