Short answer
Most AI agent pilots stall because questions about ownership, data access, security, quality measurement and operations were never answered during the pilot. A production checklist that addresses these early turns a convincing demo into an agent that runs every day with a named owner and support.
On this page
Why so many agent pilots never go live
The demo went well. The agent answered questions, drafted emails and impressed the steering committee. Then nothing happened for months. This pattern, often called pilot purgatory, is rarely caused by the AI itself. It happens because the pilot was built to prove that something is possible, not to prove that it can be run safely, at scale, by a real organization.
Typical blockers we see:
- The pilot used exported sample data; production needs live, permissioned access to systems the pilot never touched.
- Nobody was named to own the agent's results once the project team leaves.
- Security and data protection were informed at the end and now need weeks to review.
- Success was judged by impression, so nobody can say whether the agent is good enough.
- There is no plan for who monitors the agent, handles incidents or pays for running costs.
- Users were not involved, so adoption is unclear even if the agent works.
The production checklist
We use the following checklist with clients. Ideally it is started on day one of the pilot, not after the demo.
Mandate and ownership
- The agent's task, outcome metric and forbidden decisions are written down.
- A business owner is named who accepts responsibility for the agent's results.
- Human approval points are defined for actions with legal or financial effect.
Data and integrations
- Every system the agent reads from or writes to is listed, with production access requested through the normal process.
- Permissions follow least privilege and map to the existing identity management.
- Data sources are maintained by someone, with a process for updates.
Security, privacy and compliance
- A risk assessment covers failure modes such as prompt injection, wrong actions and data leakage; see AI agent risk assessment.
- Data protection documentation is complete, including legal basis and retention.
- Regulatory classification, for example under the EU AI Act, is recorded.
- Audit logging is active and reviewed.
Quality and evaluation
- A test set of realistic cases with expected outcomes exists and is rerun after every change.
- Acceptance thresholds are agreed with the business owner before go-live, not after.
- A feedback channel lets users flag wrong outputs easily.
Operations
- Monitoring covers availability, errors, quality indicators and cost.
- On-call and incident runbooks exist, including how to pause the agent.
- Change management rules are agreed for prompts, tools, models and data.
- Running costs are estimated and budgeted.
Adoption
- Users have been trained and know when to trust, check or override the agent.
- Staff representatives are informed where work processes change.
- A rollout plan moves from a small user group to full scope in stages.
Phased rollout instead of big bang
We recommend three stages after the pilot. Shadow mode: the agent runs on real cases but outputs are only compared with human results. Assisted mode: the agent produces drafts or proposals that humans approve. Autonomous within limits: the agent completes defined, low-risk cases on its own, with sampling and escalation. Each step is unlocked by meeting the agreed quality thresholds, which turns a political decision into an evidence-based one.
Getting people ready
Adoption is the most underestimated item on the checklist. Users who do not understand an agent either ignore it or trust it blindly. Through neooutput we run hands-on trainings for the people who will work with the agent and for the internal team that will extend it, always ending with working results rather than theory. For engineering teams, our agentic coding workshops cover how to build and test agents professionally.
Platform, people and operations
Many checklist items are much easier on a platform built for governed agents. kalAIDoCreate provides permissions, audit trails, SSO, integrations with SAP, CRM, SharePoint and Slack and EU data residency, so each new agent does not reinvent its foundation. Our AI Code Engineers harden pilot code and integrations through our prototype to production service, and our managed AI agents team takes over monitoring, incidents and improvements after go-live. What a service agreement for that phase looks like is described in managed AI agents with SLAs.
FAQ
What is pilot purgatory?
It describes AI pilots that prove technical feasibility but never reach production because ownership, integration, security, evaluation or operations were not addressed.
When should we start thinking about production?
On the first day of the pilot. Involving security, data protection, IT operations and a business owner early prevents long delays later.
Do we have to rebuild the pilot for production?
Not always. Sometimes the pilot can be hardened; sometimes a clean rebuild on a governed platform is faster. We assess this in a short review.
How do we know the agent is good enough?
By agreeing measurable acceptance thresholds on a realistic test set with the business owner before go-live, and tracking them in production.