Start with one workflow that costs the most manual time. Prove value there before expanding.
Why AI systems drift after launch
Response playbook when AI output quality drops.
The technical term for this is drift: the real-world data the system encounters gradually diverges from the data it was built or trained on. A new product line, a change in customer demographics, a shift in how suppliers price their goods, none of these are dramatic events on their own, but together they can move the ground under a system that was never told the ground had shifted.
Drift is rarely sudden. It tends to arrive as a slow decline in accuracy that is easy to miss week to week and obvious only once you compare this quarter’s performance to the quarter the system launched in, which is exactly why a system that seemed fine last month can suddenly look broken this month.
Rule it out first before assuming the problem is more complicated than it actually is.
The first thing to check when something goes wrong
Before assuming the AI itself is faulty, check whether the input data has changed. A surprising number of wrong decisions trace back to a data source that changed format, a field that started arriving empty, or an upstream system update that nobody flagged to the team running the AI layer.
This check takes far less time than a full investigation into the model itself, and it resolves the issue outright in a meaningful share of cases. Rule it out first before assuming the problem is more complicated than it actually is. Pull a handful of the specific decisions that went wrong and trace each one back to the exact data it was based on, rather than reasoning about the problem in the abstract.
Working on something similar?
Let's talk →How to contain the damage while you investigate
Working on something similar?
Let's talk →If a system is making financially or operationally significant wrong decisions, pause automatic execution for that specific decision type and route it to a person temporarily, rather than turning off the entire system or letting it keep running while you investigate. This limits the damage without losing everything the system does correctly elsewhere.
Communicate this pause clearly to whoever is affected by it. A temporary manual process, explained plainly, causes far less frustration than a system that continues making visibly wrong decisions while everyone waits for a fix. This kind of clear communication also builds confidence that the system is being managed responsibly, rather than ignored until someone escalates loudly enough.
When to retrain versus when to rebuild
If the underlying process has not changed but the data has shifted, retraining or recalibrating the system on more recent data is often enough. If the process itself has genuinely changed, new rules, new categories, a different workflow, retraining on old logic will not fix it. That situation calls for rebuilding the relevant part of the system around the new reality, since no amount of additional data will teach the old logic to handle a process it was never designed for.
A useful test: describe the change in one sentence. ‘Our customers behave slightly differently now’ usually points to retraining. ‘We changed how this process actually works’ usually points to a rebuild.
Preventing the next one
Ongoing monitoring that tracks the system’s decisions against actual outcomes, not just at launch but every month, is what catches drift early instead of six months after it started causing real problems. If your system does not currently have this kind of monitoring, that is worth fixing before anything else.
Set a specific, recurring date to review this monitoring, not just a vague intention to keep an eye on it. A system nobody is actively watching is a system quietly accumulating drift that nobody will notice until it becomes expensive. A short monthly check is usually enough to catch a problem while it is still small and inexpensive to fix.
Assign this monitoring to a specific person by name, not to a team in general. Responsibility that belongs to everyone tends to end up belonging to no one, and drift is exactly the kind of slow-moving problem that quietly falls through that particular gap when nobody in particular owns it.
