AI governance and compliance

AI Agent Risk Assessment Before Go-Live

AI agent risk assessment before go-live: a practical method and template covering permissions, failure modes, data, oversight and residual risk.

Short answer

An AI agent risk assessment examines what an agent is allowed to do, how it can fail, what data it touches and who intervenes when it does. Done before go-live, it turns vague worries into specific controls, an accepted residual risk and a named owner.

On this page
  1. Why classic IT risk assessments miss agent risks
  2. Our method in six steps
  3. The template
  4. Testing, not just documenting
  5. How the platform helps
  6. Fit with regulation
  7. FAQ

Why classic IT risk assessments miss agent risks

Most organizations already assess new software for security and data protection. Agents break some of the assumptions behind those templates. Their behaviour is not fully specified in code, they interpret instructions from documents and emails, they call tools with real permissions, and the same input can produce slightly different outputs. A checklist that asks "is data encrypted at rest?" is still necessary, but it says nothing about an agent that is tricked by a forwarded email into sending a supplier list to an outside address.

An agent-specific assessment fills that gap. It should be short enough to be done for every agent, and rigorous enough that an auditor, works council or data protection officer can follow the reasoning.

Our method in six steps

1. Describe the mandate

Write down in plain language what the agent owns, which outcome it is measured by and which decisions it must never make. A good mandate fits in a paragraph. If it does not, the agent is probably doing too much.

2. Map tools and permissions

List every system the agent can read from and write to, with the exact scope: which mailbox, which SharePoint libraries, which CRM objects, which API endpoints. Mark each write permission as reversible or irreversible. Irreversible actions, such as sending external messages, booking payments or deleting records, are where most of the risk lives.

3. Identify failure modes

For each tool, ask how things could go wrong:

  • Wrong output: hallucinated facts, wrong calculation, outdated source.
  • Wrong action: correct reasoning, but the wrong record or recipient.
  • Manipulation: prompt injection through documents, emails or web pages the agent reads.
  • Data leakage: personal or confidential data sent to a model, log or recipient that should not receive it.
  • Silent failure: the agent stops working or loops and nobody notices.
  • Scope creep: the agent is later given new tools without a fresh assessment.

4. Rate likelihood and impact

We use a simple three-level scale for each and discuss the ratings with the business owner. Precision matters less than a shared, documented judgement.

5. Assign controls

Typical controls for agents include least-privilege permissions, human approval for irreversible actions, allowlists for recipients and domains, input sanitization, confidence thresholds that trigger escalation, rate limits, full action logs and alerting on unusual behaviour. For personal data, anonymizing text before it reaches a model is one of the most effective controls; Textwash Pro does this locally, offline or as a privacy layer in front of ChatGPT and Claude.

6. Accept residual risk and name an owner

The assessment ends with a decision: the remaining risk is accepted by a named person with the authority to do so, with a review date tied to changes in scope. Without that signature there is no go-live.

The template

Our one-workbook template has these sections:

SectionContent
MandateTask, outcome metric, forbidden decisions
SystemsRead and write permissions per system, reversible or not
DataCategories of data, personal data, legal basis reference
Failure modesScenario, likelihood, impact, existing controls
ControlsMeasure, implemented by, how it is tested
OversightWho approves what, escalation path, response times
EvidenceWhere logs and test results are stored
DecisionResidual risk, owner, conditions, next review

We hand the template over so your teams can run it themselves for future agents.

Testing, not just documenting

A risk assessment is only credible if the controls are tested. Before go-live our AI Code Engineers run scenario tests against the agent: injected instructions hidden in documents, ambiguous requests, edge-case records and simulated outages of connected systems. Results are attached to the assessment as evidence.

How the platform helps

On the kalAIDoCreate platform, many controls from the template are configuration rather than custom work: governed agents with explicit permissions, SSO, audit trails, integrations with SAP, CRM, SharePoint and Slack, and EU data residency. That keeps assessments consistent across dozens of agents. See also how we structure audit trails for AI agents.

Fit with regulation

The assessment supports, but does not replace, a data protection impact assessment and the classification required under the EU AI Act. We link the documents so the same facts are not maintained twice. Our AI governance and compliance team runs the process with you, and our managed AI agents operations team keeps assessments current after launch, because the riskiest moment is often the quiet change three months later. For the regulatory side, read EU AI Act for AI agents.

FAQ

How long does an AI agent risk assessment take?

For a well-scoped agent, a focused workshop plus testing is usually enough. Agents with many write permissions or personal data need more time.

Who should take part?

The business owner of the process, IT security, the data protection officer and the engineers building the agent. For agents affecting employees, involve the works council early.

Do we need a new assessment when the agent changes?

Yes, whenever it gains new tools, permissions, data categories or user groups. Minor prompt adjustments can be covered by a lighter review.

Can we use the template without your services?

Yes. We hand it over so your teams can run it independently.

Tools from the jocapps ecosystem

Want this done right, and fast?

Talk to our AI Code Engineers and AI architects. In a free 30-minute call we look at your idea, prototype or roadmap and tell you honestly what it takes to ship it.

24/7 call support in English and German.

Free call

Free intro call

Tell us which task an AI agent should take responsibility for. We show how our platform, AI Code Engineers and operations plug into your stack.

  • 30 minutes, no obligation
  • Honest feasibility check
  • Concrete next step
Rather talk now? 24/7 call support in English and German+49 761 519 751 50