AI Incident Reporting: What Counts and How to Do It

A chatbot once told a customer how to commit insurance fraud. The company found out from a tweet, not from any internal channel — because nobody had defined what an “AI incident” was, nobody knew where to report one, and the support team did not think a weird chatbot answer was their problem. By the time leadership heard, screenshots had circulated for two days.

AI incident reporting is the process of recording, escalating, and learning from things AI systems do wrong. It is the feedback loop that turns failures into fixes — and in several jurisdictions, it is becoming a legal obligation. This guide covers what counts as an incident, why reporting matters, what good reports contain, and how to build the process. For the governance system it belongs to, see my overview of AI governance.

Here’s the uncomfortable truth this guide starts from: every organization running AI in production has incidents. The only question is whether you learn about them from your own process or from someone else’s screenshots. Reporting is how you make sure it’s the former.

Table of Contents

• What Counts as an AI Incident
• Why Reporting Matters More Than You Think
• The Legal Push Toward Mandatory Reporting
• What a Good Incident Report Contains
• Severity Levels That Work
• Building the Reporting Process
• What Kills Reporting (And How to Fix It)
• Learning From Incidents
• Frequently Asked Questions

What Counts as an AI Incident

Definitions vary, but a practical one: an AI incident is any event where an AI system causes harm, nearly causes harm, or behaves in a way that violates policy, law, or reasonable expectations. Concrete examples:

• Harmful outputs. The system generates instructions for wrongdoing, hateful content, or dangerous misinformation.
• Biased decisions. A hiring tool systematically disadvantages a protected group; a lending model denies qualified applicants.
• Privacy breaches. The system reveals personal or confidential data it should not have.
• Safety failures. An AI-assisted vehicle, medical tool, or industrial system malfunctions.
• Security compromises. Prompt injection, data poisoning, or model extraction attacks succeed.
• Near misses. The system almost caused harm but was caught in time. These are gold — reporting them prevents the real thing.

What does not count: the model being mediocre at its job. Poor performance is a product issue; an incident involves harm, risk, or rule-breaking.

Why Reporting Matters More Than You Think

Three reasons incident reporting punches above its weight:

1. AI failures are silent. When traditional software breaks, it usually crashes loudly. When AI breaks, it confidently produces wrong answers. Without reporting, failures accumulate invisibly until something blows up publicly.

2. Patterns hide in single events. One biased hiring decision looks like noise. Fifty, collected through reports, reveal a systemic problem. Incident databases are how industries learn — aviation safety was built on exactly this principle.

3. Regulators are watching. The EU AI Act requires providers of high-risk systems to report serious incidents to authorities. Having no reporting process is itself becoming a compliance violation.

The Legal Push Toward Mandatory Reporting

Incident reporting is moving from best practice to legal duty:

• EU AI Act. Providers of high-risk AI systems must report serious incidents — death, serious health damage, serious property or environmental damage, or infringements of fundamental rights — to market surveillance authorities, generally within tight deadlines after establishing a causal link.
• US. No general federal mandate yet, but sector regulators (finance, healthcare, transportation) increasingly expect incident reporting for AI-driven systems under their existing authorities.
• Voluntary databases. The AI Incident Database (AIID) and OECD’s AI incident monitor collect public reports. Contributing is voluntary but valuable — your incident might be someone else’s early warning.

Even where reporting is not yet mandatory, building the process now is cheaper than building it under regulatory deadline later.

What a Good Incident Report Contains

A useful incident report answers:

• What happened? Factual description of the system’s behavior. Include the actual output or decision where possible.
• When? Timestamp and duration. Was it a single event or ongoing?
• Which system? Model name, version, deployment context. “The chatbot” is not specific enough.
• Who was affected? Individuals, groups, customers — and how many.
• What was the impact? Harm caused or risk created, described concretely.
• What was the root cause? As best understood — data issue, model limitation, misuse, attack, integration bug.
• What was done? Immediate containment and longer-term fixes.
• Who reported it? So follow-up questions have somewhere to go.

The best reports are written close to the event, by the person who saw it, in plain language. A template with these fields — one page — beats a freeform essay every time.

Severity Levels That Work

Not every incident deserves a war room. A simple four-level scale:

• Critical. Actual serious harm or a live safety/security threat. Immediate escalation to leadership, containment first.
• High. Significant risk or harm to individuals, or a pattern suggesting systemic failure. Investigate within days.
• Medium. Policy violations or concerning behavior without demonstrated harm. Investigate within weeks.
• Low. Near misses and minor issues. Log, review in aggregate, look for patterns.

The key discipline: near misses get logged, not dismissed. Aviation learned that the accident you prevent is the one whose warning signs someone reported.

Building the Reporting Process

For most organizations, this is a week of work:

1. Define “incident” for your context, with examples. Publish the definition where employees will see it.
2. Create one reporting channel. A form, an email alias, a Slack channel — one obvious place. Multiple channels mean reports get lost.
3. Assign triage ownership. Someone specific reviews incoming reports, assigns severity, and routes them. A shared inbox nobody owns is a black hole.
4. Set response SLAs. Critical: hours. High: days. Medium/low: weeks. Publish them.
5. Protect reporters. Make clear that good-faith reports are welcomed, not punished. Fear kills reporting faster than anything.
6. Log everything centrally. A simple database or tracker. This becomes your institutional memory — and your evidence for auditors and regulators.

Train customer-facing teams especially: support staff see AI misbehavior first, but often do not recognize it as reportable. A 15-minute training with examples fixes that.

What Kills Reporting (And How to Fix It)

Every organization with a reporting channel discovers the same enemies:

Fear of blame. If reporters think an incident report becomes a performance review item, reporting stops overnight. The fix is structural, not rhetorical: separate incident reporting from performance management, publish a no-blame policy with leadership’s signature on it, and celebrate good reports publicly — “thanks to a support agent’s report, we caught this before it affected customers.”

Report fatigue. If nothing visibly happens after a report, people stop writing them. The fix: close the loop visibly. Acknowledge every report within 24 hours, publish quarterly summaries of what was reported and what changed, and let reporters see the trend lines their reports created.

Vague definitions. If nobody knows what counts as an incident, edge cases go unreported. The fix: concrete examples, updated annually, with a clear “when in doubt, report” rule. Over-reporting is cheap; under-reporting is expensive.

Friction. A 20-field form kills more reporting than any policy. The fix: three fields for the initial report (what happened, which system, your contact), details gathered during triage. Make the first step take under two minutes.

Learning From Incidents

Reporting without learning is bureaucracy. Close the loop:

• Root-cause analysis for high and critical incidents. Fix the cause, not just the symptom.
• Trend reviews quarterly. Are incidents clustering around one system, one team, one failure type?
• Share lessons internally — and consider contributing anonymized cases to public databases. The field learns faster when incidents are shared.
• Update controls. Every significant incident should change something: a test, a monitor, a policy, a training module.

The public incident databases deserve special mention. The AI Incident Database and similar projects collect real failure reports so the whole field can learn — the way aviation’s incident reporting transformed flight safety. Aviation’s culture is the model: near-misses are reported as diligently as accidents, reports are de-identified, and the findings reshape training and design. AI is nowhere near that maturity yet, but the organizations contributing cases today are building the shared knowledge base that will define responsible practice tomorrow. Your internal process and the public record reinforce each other; contribute to both.

Understanding how AI models are trained helps incident reviewers identify data-related root causes, and reviewing your deployed AI tools periodically surfaces systems that should be in the reporting scope.

Frequently Asked Questions