Auditing AI systems: data transparency, NIST's AI RMF and ISO/IEC 42001
An AI audit asks what data went in, how the system was tested, and who is accountable when it's wrong. Three frameworks now give companies a common way to answer.
Customers, regulators and boards increasingly want evidence that AI systems are safe, fair and well controlled. An AI audit produces that evidence. Its core questions are simple: what data trained or feeds the system, how it was tested, how it is monitored, and who is responsible.
The frameworks
| Framework | Type | What it gives you |
|---|---|---|
| NIST AI Risk Management Framework 1.0 | Voluntary US framework, released January 26, 2023 | Four functions: Govern, Map, Measure, Manage |
| NIST AI 600-1, Generative AI Profile | Companion to the AI RMF, released July 26, 2024 | Risks and actions specific to generative AI |
| ISO/IEC 42001 | International standard for AI management systems | A certifiable management system, similar in structure to ISO 27001 |
| EU AI Act | Law | Mandatory duties by risk level, including documentation and transparency |
The NIST framework is the most practical starting point for an internal audit. ISO/IEC 42001 suits companies that want a certificate to show customers. The EU AI Act sets legal requirements for companies providing or deploying AI in the EU, with general-purpose AI model duties in force since August 2025 and high-risk system rules applying from December 2027 for most uses.
What to audit
Inventory comes first: list every AI system in use, including AI features inside SaaS products, with an owner for each.
For each system, the audit then covers data, testing and operations.
Data: where training and input data came from, whether the company had the right to use it, how personal data is handled, and whether the data represents the people the system will affect.
Testing: how accuracy was measured, on what data, and whether results were checked across groups for unfair differences. For generative AI, testing should include made-up answers, harmful output and prompt injection.
Operations: who can change the system, how it is monitored after launch, how users can report problems, and what happens when it gets something wrong.
Transparency for third-party AI
Most companies buy AI rather than build it. Vendors should be able to say what data their models were trained on, at a summary level, how they test them, and how customer data is used. Providers of general-purpose AI models in the EU now have to publish a summary of their training content, which gives buyers a starting point.
What IT teams should do
- Build an inventory of AI systems and AI features in SaaS tools, each with a named owner.
- Use the NIST AI RMF's four functions as the structure for your first internal review.
- Add AI questions on training data, testing and data use to vendor security reviews.
- Record test results for each system, including checks across user groups.