By
At some point, a manager, customer, or regulator will question a specific decision made by your AI agent. If your company or team can’t explain the reasoning behind it, trust can erode quickly, and the agent(s) that once made your company efficient is now a liability.
Being able to show why a system produced a particular result is called AI explainability, and it’s quickly becoming the difference between an AI program people can trust and one that poses unexpected risk.
Banking hit the AI explainability wall first.
Last April, the Federal Reserve, the Office of the Comptroller of the Currency, and the FDIC issued revised interagency guidance on model risk management, published by the Federal Reserve as SR 26-2. This guidance replaces a 2011 standard created before most current models existed. Generative and agentic AI are outside the scope of the guidance, but its principles apply to other AI models. Regulation B has long required lenders to provide specific reasons for denying credit, and “the algorithm said no” isn’t a good answer.
The U.S. has generally addressed AI developments reactively, but there are signs this is starting to shift. In July, the ranking member of the House Financial Services Committee requested input on AI in financial services to help develop a modern federal framework that encourages responsible innovation while protecting consumers and markets.
State regulators are adopting a model bulletin from the National Association of Insurance Commissioners, which expects insurers to maintain a written program governing AI use and outlines what regulators may request during examinations. As of April 2026, 25 jurisdictions have adopted it. Internationally, the EU AI Act established a unified framework across all sectors, with obligations that scale to the system’s risk level.
Industries are converging on the same requirements. They want managers to document how their AI tools work, name who owns them, and be ready to explain a single decision months later.
As a rule of thumb, regulated industries receive these rules first, while others typically adopt similar standards later. If your company is involved with consumer finance, health, or safety, it’s best to begin preparing for AI explainability requirements now.
Take The Marshmallow Test
On the surface, moving fast looks like the best response to competitive pressure, especially when new technology promises to boost productivity. But when companies implement tools before establishing clear guardrails, departmental managers discover, test, and adopt solutions on their own. The approach accelerates progress at first, but it can also create an unmanageable patchwork of AI tools and inconsistent practices that are harder to reverse than they would have been to prevent.
Governance may at first seem like the adult version of the marshmallow test, but putting enough structure in place to move forward responsibly is what keeps the rollout from turning into chaos later. Therefore, yes, adopt technology, but do it in this order:
- Governance
- Inventory
- Evaluation
- Implementation
- Monitoring
The good news is that each step tends to reduce the cost of the next.
Governance Comes Before The Tool
First, assign ownership of the AI tool. Second, determine which data may leave your environment. Third, establish a policy for when human approval is required and how those decisions are documented.
Finally, run the exercise most governance frameworks skip. Write down what could go wrong with each tool, who it would affect, and what you would do about it. If a customer-facing assistant says something it shouldn’t, the response plan matters, and auditors treat a documented what-if as evidence that someone considered risk before it occurred.
You don’t need to start your governance plan from scratch. The AI Risk Management Framework from the National Institute of Standards and Technology is free and thorough enough for organizations of any size. The Financial Services AI Risk Management Framework tailors that approach for financial services.
An initial draft of the framework is sufficient, provided it is in place before you deploy the tools. This process should take a couple of weeks.
. . .
To see the full article, visit TechDogs, “Banking Got Its AI Explainability Rules First. Your Industry Is Next.”
