The AI feature that arrived in an update

It arrives switched on

Your CRM, your accounting package, your practice management system, your helpdesk, your HR portal — each has shipped, or will ship, an AI feature. Typically in a routine update, typically enabled by default, typically announced in a release note nobody read.

At that moment your organisation may have acquired a new recipient of its data without anybody deciding to. This is now the commonest route by which that happens: not somebody pasting into a chat window, but a checkbox that arrived ticked.

Nine questions, and what a real answer looks like

Ask the vendor. In writing.

1. Which model, from which provider, running where? "A leading large language model" is not an answer. You need the provider's name and the processing region, because everything else depends on them.
2. Is our data used to train anyone's model? The answer must be in the contract or data processing agreement, not in a marketing FAQ that can be edited on a Tuesday.
3. Is the model provider a sub-processor, and is it listed? If your organisation is a data controller under a regime like the UK or EU GDPR, adding a sub-processor normally requires notice, and often a right to object. A vendor who cannot answer this has not thought about it.
4. How long are prompts and outputs retained, and by whom? Note that this is two answers: the vendor's retention and the model provider's.
5. Can it be disabled — for the whole tenant, and per user? Get the setting's location, not a promise.
6. How is it billed? Included, per seat, or per use. Consumption pricing on a feature that colleagues will use freely is a bill nobody forecast.
7. What is logged? If a customer complains about something the feature generated, can you retrieve what it produced and when?
8. What happens when the underlying model is changed? It will be, without notice, and behaviour changes with it. Ask whether you are told.
9. What does it do when it does not know? Ask for a demonstration on a question outside your data. A feature that confabulates rather than declining is a feature that will tell a customer something untrue.

Evaluate on your own work, not the demonstration

The demonstration uses data chosen because it works.

Take twenty real items from your own queue — twenty real tickets, twenty real invoices, twenty real case notes — and run them. Then count two things: how many outputs you would have sent unchanged, and how many contained an error you had to catch.

Those two numbers settle the argument, in either direction, in an afternoon. And the second one is the number that matters, because a feature with a 5 per cent error rate on customer-facing output is not a time saving; it is a checking obligation with a subscription fee.

The default-on problem is a governance job

Someone needs to own a list: which products in your estate have AI features, whether each is on, when it appeared, and who decided.

Without that list, the honest answer to "where does our client data go?" becomes "we are not sure", and that is an answer with consequences in a regulated sector. The list does not need software. It needs one spreadsheet and one person who updates it when a release note mentions AI.

Ask whether you already have the feature

A surprising share of what these features do is already available, deterministically, in software you own.

Duplicate detection, templated responses, routing rules, scheduled reports, spell checking, merge fields, saved searches, conditional formatting. These are exact, free, already paid for, and they do not require a data protection assessment. Where the existing feature does the job, the AI version is a downgrade wearing a newer badge.

The genuine wins are the ones requiring judgement over unstructured text: summarising a long free-text history, drafting a first reply, classifying a message whose category is not deducible from any field. Those are worth the questions above. The rest usually is not.

And keep the off switch findable

Whatever you decide, know where the switch is and who can reach it. The day a vendor has an incident, or a client asks you to confirm their material is not being processed by a third party, "we would have to raise a ticket" is not the position you want to be in.