In short
- For a few weeks, the agent suggests and your team works as usual. The agent sends nothing itself.
- We compare, case by case: approval rate, errors, time saved. These figures are measured at your company, not taken from an average.
- You then decide on going live based on these figures. They also serve as the basis for calculating your return on investment.
An AI agent is a program that reads your emails, your invoices or your quote requests, then prepares an action: a draft, a classification, a reminder. Before letting it get on with it, you need to know whether it does the job well at your company, with your customers and your exceptions.
At Kendu Intelligence, this proof has a name: the shadow trial. “Acceptance testing” is the check before delivery. “Shadow”, because the agent works in the shadow of your team, without ever acting in its place.
The principle: the agent suggests, the human does, we compare
The shadow trial, or parallel run, is step 4 of our method, between building the agent and putting it into service.
The agent will be connected, through the official interfaces (APIs) of your tools where they allow it, to your mailbox, your invoicing software or your customer management. It will receive the same requests as your team and will place each suggestion in an approval screen, in preparation. It will send nothing to your customers or suppliers. It will change nothing in your software.
Your team, for its part, handles the cases as before. Afterwards, we compare what the agent suggested with what the human actually did. This is level 0 of our four autonomy levels: the agent suggests, it does not act.
A typical scenario: an SME with 30 employees wants to hand over its unpaid-invoice reminders. For three weeks, every morning the agent prepares the reminders it would send, and the accountant chases as usual. Each week, we compare the two lists. Same customers? Same tone? Did the agent plan to chase a customer in dispute, who should have been left aside?
What we measure: approval rate, types of errors, time saved
Each suggestion will be recorded in a log: what the agent read, what it suggested, what the human did, and when. We plan to draw three measures from it.
The approval rate. This is the share of suggestions that your team would have approved. Each case will be placed in one of three boxes: identical, to touch up (a phrase, a date), to redo.
The types of errors. A rate is not enough. An awkward phrase in an acknowledgement of receipt does not weigh the same as a reminder sent to the wrong customer. The gaps will therefore be sorted:
- serious error: wrong recipient, wrong amount, a commitment the agent should not have made;
- formal error: tone, wording, layout;
- missed stop: a case where the agent should have handed over (dispute, sensitive data, doubt about the sender) and did not;
- unnecessary stop: the agent handed over without need. This is caution: it costs time, not trust.
A repeated serious error weighs more than a good approval rate.
Time saved. We will record the time your team spends on the task today. Then we will measure, on a sample, the time needed to review and approve a suggestion from the agent. The difference, multiplied by the number of cases, gives the time saved.
Each agent will also have its own indicators, written in its rulebook, such as the time to first reply for an Emails and Support agent. Those that are read over several months, such as the average payment time, will be tracked during supervision.
Why not rely on an average? A study published in 2025 in the Quarterly Journal of Economics, an economics journal, followed 5,172 customer support agents equipped with an AI assistant. Requests resolved per hour rose by 15% on average. The least experienced and least skilled employees improved by about 30%. The most experienced gained a little in speed and lost a little in quality. The gain therefore depends on who does the work: only a measurement at your company tells you what you can gain.
The duration and volume needed to conclude
The shadow trial lasts a few weeks, a length set with you. It must cover a full cycle of the task: a month-end for accounting documents, at least one wave of due dates for reminders.
Volume matters as much as duration: a rate measured on 20 cases says little. The table below starts from an observed approval rate of 90%. It gives the range in which the true rate lies, with 95% confidence: a range calculated this way contains the true rate about 95 times out of 100. This is a standard statistical calculation, called a confidence interval (Wilson method).
| Cases compared | Observed rate | 95% range |
|---|---|---|
| 20 | 90 % | 70 à 97 % |
| 50 | 90 % | 79 à 96 % |
| 100 | 90 % | 83 à 94 % |
| 200 | 90 % | 85 à 93 % |
| 400 | 90 % | 87 à 93 % |
From around a hundred cases, the range becomes narrow enough to decide. If the volume is still too low at the end of the planned period, we will tell you rather than conclude on too little. You then choose: extend the shadow period, or decide knowing that the range remains wide.
The data: anonymised during the build, real during the shadow trial
During the build, we test the agent on your anonymised data. To anonymise is to make it impossible in practice, and permanently, to identify the person again. The CNIL, the French data protection authority, reminds us: once data is anonymised, data protection law no longer applies. Be careful not to confuse this with pseudonymisation, which replaces a name with a code, for example: according to the CNIL, such data remains personal.
During the shadow trial, the agent works on your real data: this is the only way to measure the real work. The safeguards are put in place beforehand:
- a contract: the GDPR (the European data protection regulation) provides that processing by a processor, that is, the provider who processes the data on your behalf, is “régi par un contrat ou un autre acte juridique” (governed by a contract or other legal act) (Article 28);
- the strict minimum: the same text requires processing only the data “nécessaires au regard de chaque finalité spécifique du traitement” (necessary for each specific purpose of the processing), that is, each aim pursued (Article 25). Connectors, the links with your software, will only have read access where possible;
- the same location as in production: your cloud account (your online IT services), preferably in a European region, your servers, or a dedicated server with us, hosted in Europe. The AI model can be operated outside the Union, under GDPR safeguards: we tell you beforehand, in the quote, where it processes your data;
- a written description: the personal data processed is described in the agent's rulebook, which you approve.
Going live: who decides, and on which figures
You decide, twice: you approve the agent's rulebook before it is built, then you give the green light to go live after the shadow trial.
This green light rests on the shadow trial measurements. The indicators to track are in the rulebook approved at the start. It is better to write the level to reach there too: setting the bar before seeing the results avoids moving it afterwards.
Going live is gradual. At level 1, the agent prepares and your team approves each action. Level 2, where it acts alone on safe cases defined in writing, only comes on the basis of the measurements and with your written agreement. Some actions never go beyond level 1: a payment, sending a document that commits you, deleting data. After a serious alert, the agent drops back a level.
The shadow trial figures are also used to calculate your return on investment:
- monthly gain = hours saved per month × hourly cost of the people concerned (salary and employer contributions);
- monthly cost = monthly subscription (on quote);
- payback period = set-up cost (on quote) ÷ (monthly gain − monthly cost).
If the monthly gain does not exceed the monthly cost, there is no return on investment, and the calculation shows it. We do not publish an average gain: we are starting out, and we do not yet have enough measurements to do so honestly. Your calculation is done with your figures.
And if the results are not good enough?
This is what the shadow trial is for: finding the problem before it reaches a customer.
- Fix and replay. A gap can come from a missing rule or an unforeseen case. We fix it, publish a new version of the agent and restart the shadow trial on new cases.
- Reduce the scope. We keep the cases where the agent is reliable, for example acknowledgements of receipt, and your team keeps the rest.
- Stay at level 1. The agent prepares, the human approves everything. The gain is smaller: the shadow trial measurements tell you whether it covers the monthly cost.
- Stop. If the agent does not bring enough, it is better to know after a few weeks than after six months. The conditions for stopping at this stage will be stated in your quote.
A useful agent knows when to hand over. The shadow trial is the moment when you check that it does so at the right time.