“Soon, AI will have a systemic impact on the entire financial system.” | Image credit:
Derek Fenech/CCN
Share
Key Takeaways
Black-box AI undermines regulated finance. Institutions cannot claim accountability if they cannot inspect how their AI systems make decisions.
Autonomous agents are risky, but opacity is the systemic threat. The real danger is not just AI autonomy, but the inability to audit its logic.
Transparency changes the accountability equation. Open-source AI allows regulators, auditors, and compliance teams to examine models directly and identify bias or failure.
Finance requires inspectable infrastructure. Just as critical internet systems rely on open foundations, AI in banking must meet the same standard of scrutiny.
European regulators are finally sounding the alarm on AI in banking, but they’re focused on the wrong threat.
Major financial institutions are embedding artificial intelligence across their operations, giving it more autonomy than ever. Yet, global regulators are increasingly aligned in their concerns.
In mid-December 2025, the British Financial Conduct Authority warned that autonomous AI agents pose a significant risk to financial stability.
Benchmarking real-world reasoning is step one. Most models fail silently.
Because agentic AI models can run independently and execute tasks with real-world impact, they are already being deployed in credit scoring, fraud detection, risk management, and market surveillance.
Critically, regulators warn that these models may soon outpace existing oversight and control mechanisms.
AI decisions that impact real people must be transparent and explainable. However, the closed-source models currently in wide use fall short.
Bet on ESports with These Partners
Sponsored
Disclosure
We sometimes use affiliate links in our content, when clicking on those we might receive a commission at no extra cost to you. By using this website you agree to our terms and conditions and privacy policy.
AI hallucinations have made headlines since the launch of OpenAI’s GPT-3, which ignited the explosion of interest in this technology.
Because these errors stem from a model finding patterns where none exist – often due to biased training data or issues in the learning process – they are notoriously difficult to eliminate.
While dangerous in general use, these mistakes are catastrophic in finance, especially regarding autonomous agents with limited oversight.
Agentic AI models could freeze users’ accounts, deny them loans, or set prices arbitrarily.
This is not just theoretical. Consider 2019, when Apple and Goldman Sachs came under scrutiny for their AI models showing gender bias in Apple Card’s credit approvals. Worse, their engineers didn’t spot the error until users did.
Today, financial institutions are in a more precarious position. The models they use are proprietary black boxes, with their training data and decision tree logic guarded as trade secrets.
Financial institutions are now making trillion-dollar decisions based on outputs they cannot inspect, audit, or verify. They and their clients must trust that the model works as advertised, with no way to confirm it.
Even more concerning: soon, AI will have a systemic impact on the entire financial system.
Why Transparency Isn’t Optional in Regulated Markets
In late 2025, the Dutch National Bank issued a warning that artificial intelligence is deepening Europe’s dependence on a handful of foreign tech firms and creating systemic risks that could hobble payment systems or spike costs across the continent.
Global regulators echoed this concern. The European Central Bank flagged the risks that come from outsourcing cloud and AI operations to third parties, while the Bank of England warned about financial stability implications.
While these measures are important, they miss a key risk in third-party dependence.
Closed-source AI models are increasingly at odds with regulated finance, no matter where their owners are located geographically.
TheFinancial Stability Board has flagged the risks of relying on opaque training data and external AI providers. In the United States, Acting Comptroller of the Currency Michael J. Hsu has warned that such systems can create a diffusion of responsibility that weakens accountability and trust.
This supports a deeper conclusion: If financial institutions cannot examine how a model reaches its conclusions, neither can auditors, compliance teams, or regulators. Consequently, the accountability framework breaks down.
In closed-source models, hallucinations are unauditable. When a proprietary model produces a credit decision or a risk assessment, there’s no audit trail into the reasoning and no way to identify whether the output reflects genuine analysis or fabricated confidence.
Why Open-Source AI Fits Regulated Finance Better
Open-source AI offers a necessary alternative. By making architectures, training processes, and evaluation methods visible, open models provide the transparency that closed systems lack. It doesn’t mean they don’t hallucinate; it means those hallucinations are easier to spot.
Researchers, auditors, and regulators can examine for hidden biases and unsafe behaviors. They can audit how these systems think.
The point isn’t that open models are “good” merely because they are open; it is that their risks are legible.
With public visibility, users and the public can hold AI firms accountable. It discourages companies from unethical data collection, while making harmful outputs easier to spot.
Furthermore, a global community of developers and academics can stress-test models and identify vulnerabilities faster than any single company’s team.
“When the logic is visible, the biases can be identified, and the decision path can be reconstructed.”| Image source: Himanshu Tyagi
Open-Source AI Strengthens Systemic Resilience in Finance
That “many eyes” effect matters because the most dangerous failures in AI aren’t always obvious in a controlled demo.
Instead, they show up at the edges, under adversarial pressure, or in the hands of users doing unexpected things.
Open-source models can implement the same safeguards as closed systems, like filters, monitoring, and usage restrictions, but without requiring blind trust in corporate auditing.
Open-source AI doesn’t eliminate risk, but it changes the accountability equation by allowing regulators to examine models directly.
Institutions can test, adapt, and fine-tune them for specific use cases. Most importantly, when something goes wrong, there’s a forensic trail.
Transparent AI as Critical Financial Infrastructure
This principle of transparency has provided the bedrock for the evolution of critical infrastructure across every other domain. Linux runsmore than 90% of the world’s web servers, including those operated byGoogle,Microsoft, and major financial institutions.
The stability of the internet rests on open-source foundations that can be inspected, verified, and improved by anyone.
AI in finance needs the same treatment. When the European Central Bank evaluates a bank’s risk models, it should examine the methodology. When an institution deploys an open-source AI system, the same standard of scrutiny becomes possible.
When the logic is visible, the biases can be identified, and the decision path can be reconstructed.
Rebuilding Trust Through Transparent AI
There is risk, but the solution requires changing the relationship between financial institutions and the AI systems they depend on.
That means moving toward models that provide transparency, so institutions retain clear ownership of decision-making.
This allows regulators to perform genuine oversight rather than accepting assurances from vendors with every incentive to obscure their methods.
AI adoption in finance has reached a scale where these questions can no longer be deferred. Closed black boxes cannot meet that standard. Open, decentralized systems can.
Disclaimer:
The views, thoughts, and opinions expressed in the article belong solely to the author, and not necessarily to CCN, its management, employees, or affiliates. This content is for informational purposes only and should not be considered professional advice.
About the Author
Himanshu Tyagi
Himanshu Tyagi is a professor of engineering at the Indian Institute of Science (IISc), Bengaluru, where he teaches information theory and machine learning. He is also a co-founder of Sentient, a company building open-sourced artificial intelligence models, and an advocate for decentralized AI.
