Explainable AI: Looking Inside a Decision

Explainable AI is the effort to make a model’s decisions understandable to the people affected by them. A bank declines a loan, a hospital system flags a scan, a hiring tool ranks candidates: in each case someone will ask why. Many modern models cannot answer in a way a person can follow, because their behaviour comes from millions or billions of numbers that no one can read like a rulebook. Explainable AI, often shortened to XAI, is the set of methods and design choices that try to bridge that gap.

This guide explains explainable AI in plain words. It covers why models are called black boxes, the difference between models that are interpretable by design and explanations added afterwards, the main methods, what a good explanation must and must not claim, the special case of language models that write their reasoning, how regulation uses the idea and how to ask for explanations in your own work.

Why explanations matter

Explanations are not only a courtesy. They serve several practical purposes.

Trust and use. People rely on a system more appropriately when they understand what it does well and where it fails.
Debugging. If a model is right for the wrong reasons, such as recognising a watermark instead of a disease, an explanation can reveal it before it causes harm.
Fairness. Seeing which inputs drive decisions helps detect proxies for protected characteristics. Our article on AI bias describes how such patterns arise.
Accountability. When a decision affects someone’s life, they need reasons they can challenge.
Learning. Experts can learn from what a model has found, or reject it when it conflicts with domain knowledge.

The broader field is summarised in the overview of explainable artificial intelligence.

What a black box is

A black box is a system whose internal workings are not understood well enough to explain individual outputs. Deep neural networks are the classic example. They learn patterns from data by adjusting enormous numbers of parameters, and the result works well but does not come with a readable list of rules. You can see inputs and outputs, and you can measure accuracy, but you cannot easily say why this applicant was scored 0.62 and that one 0.71.

Not every model is a black box. A short decision tree, a simple scoring table or a linear model with a handful of inputs can be read directly. The trade-off is that the simplest models are sometimes less accurate on complex data, though for many tabular business problems the gap is small. A useful first question for any project is whether a simple, interpretable model would be good enough, because then no extra explanation machinery is needed.

Approach	How it explains	Strength	Weakness
Interpretable by design	The model itself is readable, such as rules or a small tree	Faithful to how decisions are made	May be less accurate on complex data
Post-hoc explanation	A separate method describes a complex model’s behaviour	Works with almost any model	An approximation; can mislead
Example-based	Shows similar past cases or counterexamples	Intuitive for non-experts	Depends on the quality of the examples
Documentation	Describes data, purpose and limits in plain words	Useful to every audience	Does not explain individual decisions

Explainable AI methods in plain words

Several families of techniques give you a view into a model’s behaviour. You do not need the mathematics, only the idea of what each tells you.

Feature importance. Ranks which inputs matter most overall, for example that income and payment history drive a credit model more than age. Useful for a global picture.
Local explanations. Describe one decision. The method known as LIME, introduced in the paper “Why Should I Trust You?”, fits a simple model around one prediction to show which inputs pushed it up or down.
Shapley-based explanations. The approach called SHAP, described in a paper on interpreting model predictions, shares credit among inputs using ideas from game theory, giving each a contribution to the result.
Saliency maps. For images, highlight the pixels that most influenced the output. They look convincing and can be unreliable, so treat them as hints.
Counterfactual explanations. Say what would have to change for a different outcome, such as “the loan would have been approved with an income 10 percent higher”. They suit people who need to act on a decision.
Probing and interpretability research. Studies the internal representations of a network to find what it has learned, and is an active research area for large models.

Each method answers a different question. Pick the one that matches the audience: a regulator may want global behaviour, a customer wants a counterfactual and a data scientist wants local details for debugging.

A worked example in words

Imagine a small lender that uses a model to score loan applications. The overall feature importance shows that payment history and the ratio of debt to income matter most, which matches what credit officers expect. That is a reassuring global picture. Then an applicant is declined and asks why. A local explanation shows that two late payments last year pushed the score down more than any other factor, and a counterfactual explanation says that the application would have passed with a lower requested amount.

Now compare a worrying case. The same tool shows that the applicant’s postcode has a large effect, even though postcode was never meant to matter. That finding does not prove discrimination, but it is a signal to investigate, since postcode can act as a proxy for ethnicity or income in some places. Without any explanation method, nobody would have noticed. The example shows the practical value: explanations turn a vague worry into a specific question that a person can answer.

What makes an explanation good

A flashy chart is not an explanation unless people can use it. The US National Institute of Standards and Technology has published four principles of explainable artificial intelligence, which are a helpful checklist.

Explanation. The system provides reasons for its outputs.
Meaningful. The explanation can be understood by its intended audience.
Explanation accuracy. It correctly reflects how the system produced the output.
Knowledge limits. The system operates only within the conditions it was designed for, and signals when it is outside them.

The third principle is the one most often broken. An explanation can be persuasive and wrong, for example a tidy story that does not match what the model actually computed. A good explanation also says what it does not show. Saying “income was the strongest factor” does not mean that income caused the outcome in a causal sense, and explanations of statistical models describe patterns in the data, not reasons in the world.

Explaining language models

Large language models add a twist. They can write an explanation in fluent text, and it is tempting to read it as a view into the model’s thinking. Research on this question suggests caution: a written chain of reasoning may help the model reach better answers and help you review its work, but it is not guaranteed to be a faithful record of the computation that produced the answer. A model can give plausible reasons that differ from the true factors. Our articles on chain-of-thought prompting and reasoning models describe how step-by-step output is used.

That does not make it useless. A written explanation is a good thing to review for errors, missing steps and unsupported claims, and asking for sources lets you check the evidence directly. But for high-stakes decisions, do not treat a model’s own account of itself as proof. Verify the facts it cites, and keep a person accountable. See also our guide to AI hallucinations for why fluent text is not evidence.

Explainability and the law

Several rules touch on explanation. Under the European Union’s GDPR, people subject to certain automated decisions are entitled to meaningful information about the logic involved. The EU AI Act sets transparency and documentation duties for high-risk systems and gives affected people certain rights to explanations of decisions made with them. Our article on the EU AI Act explains the risk categories, and the guide to a data protection impact assessment for AI tools shows how to document risks and mitigations.

Law differs between countries and sectors, such as credit, insurance, health and employment, and you should confirm your obligations with a qualified adviser. A practical principle holds everywhere: if a decision affects a person significantly, be ready to explain it in words they can understand and to offer a route to challenge it.

How to ask for explainability in your own work

You do not need to build models to apply these ideas. If you buy, deploy or approve AI tools, you can ask good questions.

What is the purpose? What decision does the system support, and who is affected?
What data does it use? Which inputs matter most, and are any sensitive or likely to act as proxies?
How is it explained? What does the vendor show for one decision, and is it faithful to the model?
How is it tested? Accuracy overall and across groups, and behaviour outside its design range.
Who can override it? Is there a human with authority, and how are overrides recorded?
What happens when it is wrong? Is there an appeal route and a way to correct the data?

Our guides to evaluating an AI assistant and to AI benchmarks show how to test claims, and the article on AI guardrails covers limits that sit around a model.

Using an assistant to explain an AI system

An assistant can help you understand explainability itself. Ask it to explain SHAP with an analogy, to turn a feature importance table into a plain-language summary for a customer, to draft a counterfactual explanation that a person could act on or to list the questions to ask a vendor. Check technical claims against sources, because a model can describe a method confidently and still make a mistake in the details.

Mio is one assistant with Chat, Write, Research, Code and Design modes, and you never pick a model: Mio routes each request to one suited to the task. Chat mode explains concepts at the level you choose, Write mode drafts customer-facing explanations from facts you provide and Research mode on paid plans searches the web and cites sources, so you can read the original papers. Mio does not look inside other companies’ models, and it does not replace a technical audit. Your chats and files remain yours, they are not used to train models and you can export or delete them at any time.

Frequently Asked Questions

What is explainable AI in simple terms?

It is a set of methods and design choices that make AI decisions understandable to people. Some models are readable by design, and for complex ones, extra methods describe which inputs mattered or what would change a result. The goal is explanations that are meaningful to the audience and accurate about how the system works.

Why are some AI models called black boxes?

Because their behaviour comes from many learned parameters that cannot be read as rules. You can observe inputs and outputs and measure accuracy, but you cannot easily say why one individual case got a particular result. Deep neural networks and large language models are common examples.

Are SHAP and LIME reliable?

They are useful tools, but they give approximations and can be sensitive to settings and correlated inputs. Treat them as evidence to investigate, not as proof of cause. Check that the explanation is stable across similar cases, and combine it with domain knowledge and testing of the model’s real behaviour.

Does an AI’s written reasoning count as an explanation?

It can be helpful to review, but it is not guaranteed to reflect how the model reached its answer. A model can write plausible reasons that differ from the true factors. Use written reasoning to spot errors and missing steps, verify the facts it cites and avoid treating it as a full account for high-stakes decisions.

Do I need explainable AI for everyday tools?

For low-stakes uses, such as drafting an email, you mostly need to review the output. The need grows with the impact on people: lending, hiring, health, insurance and public services deserve explanations, testing and human oversight. Ask vendors what they provide, and prefer simpler interpretable models where they perform well enough.

The Bottom Line

Explainable AI is about giving people reasons they can understand and trust, without pretending that every model can be fully opened up. Use interpretable models where they are good enough, treat post-hoc methods as helpful approximations, check explanations for accuracy and keep humans accountable for decisions that matter. To explore the ideas with a patient study partner, start on the free plan and compare options on the Ask Mio pricing page.

Explainable AI: How We Look Inside a Model’s Decision