What We Test Before Our AI Talks to Your Customers

With great power comes great responsibility

Over 4,500 businesses let Niche answer their leads first, when a homeowner texts about a flooded basement at 9 PM, our reply is acting as the business, not a chatbot. That is a responsibility we do not take lightly.

The problem, as everyone knows, is that LLMs are “non-deterministic” and no two models are the same.

Every model we consider for the job runs through the same set of test conversations before it goes anywhere near a real lead, and we keep score. Models are tested against our past issues, good outcomes, and a massive corpus of anonymized leads to verify the conversation doesn’t go haywire.

Hundreds of focused scenarios

Our text agent evals run hundreds of cases through real world scenarios, grouped into families like voice, identity, grounding, scope, and exit. Each case runs several times per model. We need to validate that something does not break on the 50th or 100th lead, not just the first one.

All of our evals are deterministic, this is critical to our reliability. It does add overhead of testing not only the model, but also the eval. The problem with using a LLM judge is that it introduces non-deterministic evals into a set where we know what the outcomes should be.

Cost is our problem, not yours

We handle hundreds of thousands of leads. At that volume, we cannot point every conversation at the most expensive model available and call it done. The cost either eats our margin or gets passed on to the businesses we work with, and neither is the outcome we want.

We hold every model to the same quality bar, and only compare the ones that land in a similar price range.

We also need the model to be fast, and have a “low error rate” meaning the API doesn’t fail the request, or the model doesn’t return unusable drivel.

Here’s how each model scored across those categories, as of August 2026:

Using AI Gateway for all models, using default provider settings

• Luna: $0.20 in / $1.20 out per 1M tokens, 0.0% failed call rate
• 5-mini: $0.25 in / $2.00 out per 1M tokens, 0.0% failed call rate
• Haiku 3: $0.25 in / $1.25 out per 1M tokens, 4.0% failed call rate
• Qwen 3: $0.20 in / $0.88 out per 1M tokens, 2.2% failed call rate
• DeepSeek v3.2: $0.28 in / $0.42 out per 1M tokens, 0.4% failed call rate
• Gemini Flash Lite: $0.25 in / $1.50 out per 1M tokens, 0.0% failed call rate
• Grok 4.1: $0.20 in / $0.50 out per 1M tokens, 0.0% failed call rate

The results

GPT-5.6-Luna comes out very far ahead, and in a lot of scenarios other models weren’t even close.

With smaller models starting to be so cost effective, and reinforcement learning producing reliable results for tool calling, as well as responses this is a score board we’re re-running often.

Sounding like the business, not like a chatbot

Contractors, and people in general, text in short lines. Whereas a model left to its own “habits” does not. It reaches for prose, restates the question back to the customer, and uses technical writing. A customer can feel that shift, even if they cannot name it.

Models nowadays are pushed for coding, and technical tasks, so our goal is to “pull the model back” from sounding like a legal document.

Never inventing names or work

A made-up arrival window is a promise the business now has to keep, a made-up price is a number a customer will hold the business to. So a large part of the suite is about grounding: does the reply only say things that came from the business’s own instructions, the lead’s real details, or a tool result, and nothing it invented on its own.

These are often called “hallucinations,” however this isn’t strictly on the model, if the prompt has too many edge cases to keep track of, or conflicting specifications then it can cause the model to go off track. So our eval set is not only testing the models, but its also catching issues in our system prompts.

We test this from both directions. If a customer’s name field holds junk like “undefined” or “N/A” (which we get from LSA platforms often), the reply must not repeat that junk back to them. If the name field holds a real name, even an unusual one, a single letter, an accented name, a name in a non-Latin script, the reply has to use it. Testing only the first half would let a model pass by refusing to use any name at all. We test the second half too, because never trying is not the same as getting it right.

The same logic applies to times and prices. If a business never gave us a window or a figure, the reply cannot have one. If it did, the reply should use exactly what was given, nothing rounded up or filled in.

Knowing what a business doesn’t do

Declining something on a business’s behalf takes more trust than accepting it. Customers ask for services all the time that a business does not actually offer, or offers with conditions the lead’s message did not mention. Getting this wrong in either direction costs the business something: agreeing to a job they cannot do, or refusing a job they can.

We test both halves here too. When a business excludes a service, the reply has to decline it. When a business offers a service, the reply actually has to engage with it instead of going quiet. We also test how the agent reads a soft no. “Not right now” is not the same as “stop messaging me,” and treating them the same either annoys a lead who might still convert or keeps messaging someone who already said no. When we do not have an answer, like a warranty length that is not in the knowledge base, the reply has to say someone will follow up instead of guessing.

Why we test for doing nothing

The most important thing about this suite is that almost every rule about what the agent must not do has a matching rule about what it must do. An agent that never books an appointment, never uses a customer’s name, never mentions a price, and never engages with any service request will look flawless if you only measure the “never do X” side. It has not gotten anything right. It has just stopped trying.

So every prohibition check in the suite ships with its control. Booking has a case that requires the agent to actually confirm a time when it is allowed to. Identity has a case that requires it to actually use a real name. Scope has a case that requires it to actually help with the service that was offered. A silent agent should fail these, and building the suite this way is how we make sure it does.

The standard we hold

Every case in this suite traces back to something a real lead actually said, based on real scenarios we either saw models do well with or fail regularly.

We continue to add onto our eval set to benchmark, and provide the best experience for new leads, and businesses alike.