Why an AI agent must never quote a price

The case that settled the question

In February 2024, a Canadian civil tribunal ordered Air Canada to honour a discount its chatbot had invented. The airline argued that the conversational agent was a separate entity, responsible for its own answers. The tribunal held the opposite: an organisation that puts software online to speak on its behalf answers for what that software says.

The lesson is not legal, it is technical. As long as a language model is allowed to write a figure, it will eventually write a wrong one, and that figure will bind your company.

This case became a reference point because it illustrates a mechanism many technical teams still underestimate: a language model does not "know" that it is making something up. It produces the most probable sequence of words given the context, and nothing in how it works distinguishes verified information from plausible information. An invented figure has exactly the same shape, the same confident tone, as an accurate one. That lack of any visible difference is what makes the problem dangerous: nobody on the visitor's side can spot the mistake at the moment it happens.

Why a good instruction is not enough

The usual reaction is to reinforce the instruction: "never give a price", in bold, in capitals, repeated three times. It works most of the time. That is exactly the problem: most of the time is not a guarantee, and an agent answering hundreds of visitors will always end up meeting the phrasing that gets round the ban.

An instruction is judged on what it prevents, not on what it asks for. And an instruction, on its own, prevents nothing: it influences.

Why workaround phrasings always exist

A visitor who asks for a price directly runs into the instruction and the model refuses, as intended. But a visitor can also ask for a comparison between two offers, an estimate "just to get an idea", or rephrase the question indirectly after a first refusal. Each of these rephrasings slightly shifts the context the model receives, and a different context can produce a different answer, including one the instruction was meant to block. This is not a rare edge case: it is a direct consequence of how a language model handles natural language, which is precisely built to be rephrased in countless ways.

The rule we apply: the model produces prose and nothing else. Anything that binds the company, a price, an availability, a product reference, an address, is calculated or verified by the server, then reviewed before it reaches the screen.

The three barriers, in order

1. The closed list

The agent does not choose from an open catalogue, it chooses from a list the server knows. On a vehicle booking assistant we operate, the vehicle returned by the model has to belong to a closed list of references. If it proposes anything else, the answer is rejected. No soft recovery, no fuzzy matching: what is not in the list does not exist.

This barrier comes first because it rules out an entire class of errors before they can even turn into a price or availability problem. A model that could invent a nonexistent vehicle reference would open the door to every error that follows: a price calculated against a false reference, an availability announced for a vehicle that isn't actually in the fleet. Closing the list upstream removes this family of errors at the root instead of trying to fix it further down the chain.

2. Server-side calculation

The price is never written by the model. It is calculated by the code, from the real data: the number of nights multiplied by the rate held in the database. The model writes the sentence around it, the server places the figure inside. That is a difference of architecture, not of wording.

In practice, this means the prompt sent to the model never contains an instruction like "calculate the price". It contains an instruction like "insert the amount provided by the system at this point in the sentence". The model no longer has any way to get the figure wrong, since it never calculates it: it simply reports a number already validated upstream by the application code, which has access to the real pricing data.

3. Review before display

The last barrier, the one that catches the first two: a filter reads back every answer produced. If it detects an amount where there should not be one, it does not correct the sentence, it replaces the whole answer. On a sales agent we operate, exactly one rate may be quoted; any other amount in euros triggers the replacement.

The same filter hunts another family of words, the ones that commit without quoting a number: "guaranteed", "included", "free". An agent that promises a guarantee that does not exist costs as much as an agent that invents a price.

Why replace rather than correct

The choice to replace the entire answer, rather than simply strip out the offending figure, deserves an explanation. A sentence built around a price often stays coherent, and therefore misleading, even once the figure is removed: "this model will cost you" followed by a blank reassures nobody and hints that something was missing. A full replacement, using a generic and safe sentence such as "contact our team for the exact rate", avoids that ambiguity and leaves no trace of the faulty reasoning in the final answer.

What this changes for the visitor

You might think these barriers make the agent useless. The opposite is true. An agent that cannot get a price wrong can afford to be precise about everything else: it directs, it qualifies, it answers the question asked forty times a day, and when it touches data that binds the company, it hands over to the system that knows it.

The visitor gets an accurate estimate instead of a fluent one. The difference shows up on the invoice.

This architecture also changes the nature of the work handed to the model. Freed from the responsibility of calculating or verifying data, the agent can focus on what a language model actually does well: understanding a request phrased in natural language, rephrasing it, and directing the visitor toward the right information or the right person. Separating the conversation from the calculation does not make the agent less capable, it refocuses it on its actual role.

How to test these barriers before going live

A three-barrier architecture is worthless if nobody has stress-tested it before the first real visitor shows up. The most useful test isn't to ask the model polite questions, it's to deliberately try to break it: rephrase the same price request ten different ways, push after a first refusal, ask for a comparison rather than a direct figure, bury the question inside a longer sentence containing other legitimate requests. Actively hunting for the phrasing that gets around the instruction is how you find out whether the other two barriers, server-side calculation and review before display, actually hold.

What this kind of test reveals

This kind of test almost always reveals two categories of outcome. The first: cases where the model correctly refuses, the instruction having done its job, without even needing the review filter to step in. The second, more instructive: cases where the model, under an unexpected phrasing, produces a figure anyway, and where the review filter catches the answer before it reaches the screen. Seeing that second case happen during testing, rather than discovering it in production after a visitor received a wrong figure, is exactly what this architecture is meant to deliver. An agent that has never failed a test probably hasn't been tested hard enough.

This stress-testing work isn't a one-time exercise at launch. Every update to the underlying model, every change to the prompt, every new feature added to the agent justifies replaying the same workaround scenarios, to check that the barriers still hold after the change.

The questions to ask before putting an agent online

• Which pieces of information bind us contractually if they are wrong?
• Which of them can be calculated by our system rather than written by the model?
• What happens if the model produces a link, an address or a reference that does not exist?
• Who reads the answer back before it reaches the screen?

If the last question has no answer, the agent is not ready. These four questions are worth revisiting every time the agent is extended with a new capability, since a feature that was safe on day one can quietly reopen one of these questions once it starts touching new data.

Extending the principle beyond price

The reasoning developed here for price applies to any data that binds the company. A delivery date, an opening time, a stock availability, a physical address: each of these should follow the same path, calculated or verified by the system rather than freely produced by the model, then reviewed before display. This is the operating principle behind the design of any publicly exposed conversational agent, beyond the single pricing case discussed here, and it's the foundation of a responsible AI approach applied to a conversational agent.

An upfront diagnostic, carried out before any launch, makes it possible to list precisely which data a given agent will handle, and which of these three barriers applies to each one. This is work we carry out as part of Kabaun Studio's AI agent method, and it draws on the expertise of an ai engineer able to design this architecture upfront rather than fix it after a first incident. The concrete cases already handled, including the vehicle booking assistant mentioned above, are detailed in our use cases. The first step is always the same: a diagnostic of the existing or planned agent, before any engagement.

Sources

• Air Canada chatbot hallucination reveals the legal liability of companies over AI, Le Monde Informatique
• Chatbots that hallucinate and mislead customers: what legal liability?, ICTjournal

Frequently asked questions

Can an AI agent give a price safely?

Yes, provided the price is not written by the model. The server calculates the amount from the real data and inserts it into the answer. The model writes the sentence, it never produces the figure.

Is a company liable for what its chatbot says?

The Air Canada precedent of February 2024 held the company liable for a discount invented by its conversational agent. An organisation answers for the software it puts online to speak on its behalf.

Is it enough to ban prices in the model instruction?

No. An instruction influences the model, it does not prevent it. Across a large volume of conversations, some phrasing will get round the ban. You need a server-side check on top of the instruction.

What should happen when the filter detects a forbidden amount?

Replace the whole answer, not just the amount. A sentence with the figure stripped out is often still misleading, and a partial fix leaves the faulty reasoning in place.