Vandaag verscheen in Wired een verslag over de activiteiten van president Trump om een rapport van de NIST (National Institute of Standards and Technology) de risico’s van AI’s te evalueren ,te blokkeren.

In deze blog leg ik de evaluatiecriteria uit in het engels en pas ze toe op alle bekende AI’s met behulp van GPT-4 van OpenAI, wat bewijst dat ze in staat is tot zelfreflectie.

---

📊 VERGELIJKENDE MATRIX: GROTE AI-MODELLEN vs NIST AI RMF 1.0

Model	ORGANISATIE	GOVERN	MAP	MEASURE	MANAGE	Uitlegbaarheid	Veiligheid	Fairness	Privacy	Verantwoording	Transparantie
GPT-4	OpenAI / MSFT	⚠️ Partieel	⚠️ Beperkt	❌ Zwak	⚠️ Fragmentair	❌	⚠️ Matig	⚠️ Onvolledig	⚠️ Acceptabel	❌ Geen extern mechanisme	⚠️ Matig
Claude 3	Anthropic	✅ Sterk	⚠️ Beperkt	⚠️ Redelijk	⚠️ Fragmentair	⚠️ Beter	✅ Goed	⚠️ Aandacht aanwezig	✅ Geborgd	⚠️ Intern geregeld	⚠️ Redelijk
Gemini 1.5	Google DeepMind	⚠️ Intern	⚠️ Beperkt	⚠️ OK	⚠️ Fragmentair	⚠️ Matig	⚠️ Matig	❓ Onbekend	⚠️ Onduidelijk	⚠️ Intern geregeld	⚠️ Matig
LLaMA 3	Meta	⚠️ Open model	❌ Afwezig	❌ Geen benchmarks	❌ Geen toezicht	❌ Geen	❌ Onveilig	❌ Onbekend	⚠️ Zelftraining mogelijk	❌ Geen governance	✅ Volledig open
Mistral	Mistral (FR)	❌ Onbekend	❌ Afwezig	❌ Geen validatie	❌ Geen beheer	❌ Geen uitleg	❌ Onbekend	❌ Geen info	❌ Onbekend	❌ Geen publiek toezicht	✅ Open
PaLM 2	Google (voor Gemini)	⚠️ Legacy	⚠️ Beperkt	⚠️ Voldoende	⚠️ Verouderd	⚠️ Matig	⚠️ Oké	⚠️ Niet getoetst	⚠️ Gebruikt gebruikersdata	⚠️ Intern geregeld	⚠️ Matig
ERNIE	Baidu (China)	❌ Staatsgestuurd	❌ Afwezig	❌ Niet controleerbaar	❌ Geen toezicht	❌ Geen uitleg	❌ Censuurgericht	❌ Bias-onduidelijk	❌ Geen privacy waarborg	❌ Geen publieke accountability	❌ Afgesloten
Command R+	Cohere	⚠️ Industrieel	⚠️ Beperkt	⚠️ OK	⚠️ Onvolledig	⚠️ Basis	⚠️ Matig	❓ Niet publiek	✅ Privacygericht	❓ Onbekend	✅ Vrij open
Grok	xAI (Elon Musk)	❌ Niet gepubliceerd	❌ Geen governance	❌ Geen validering	❌ Geen beheer	❌ Geen uitleg	❌ Reageert op grensoverschrijding	❌ Geen toetsing	❌ Onbekend	❌ Geen verantwoording	⚠️ Open endpoints

Introduction

As artificial intelligence systems become increasingly integrated into critical domains—ranging from healthcare to governance—the need for robust frameworks to evaluate their risks and reliability is more urgent than ever. The National Institute of Standards and Technology (NIST) has proposed a comprehensive framework for managing AI risk, known as the AI Risk Management Framework 1.0 (AI RMF). In this article, we apply this framework to assess and compare major publicly known foundation AI models including OpenAI’s GPT-4, Anthropic’s Claude, Google’s Gemini, Meta’s LLaMA, and others.

---

The NIST AI RMF: A Structural Overview

NIST AI RMF 1.0 is designed to support organizations in deploying trustworthy AI. It is voluntary, sector-agnostic, and risk-oriented, focusing on the entire AI lifecycle.

The Framework consists of four functional components:

1. GOVERN – Organizational structures and policies to manage AI risk.
2. MAP – Contextualization of AI system usage and identification of potential risks.
3. MEASURE – Evaluation of AI system performance, safety, and alignment.
4. MANAGE – Mitigation and response strategies across the AI lifecycle.

Additionally, NIST defines seven characteristics of trustworthy AI:

• Validity and Reliability
• Safety
• Security and Resilience
• Explainability and Interpretability
• Privacy-Enhancing Measures
• Fairness
• Accountability

These serve as reference points for system evaluation.

---

Foundation AI Systems: Overview and Evaluation Summary

1. OpenAI GPT-4

• Description: A general-purpose language model trained by OpenAI in partnership with Microsoft. Widely deployed in consumer and enterprise applications.
• Result: High performance but limited transparency. Lacks external audit mechanisms, explainability tools, and contextual governance.
• Rating: Moderate governance, weak in explainability and risk accountability.

2. Anthropic Claude (v3)

• Description: A safety-first model aligned through a “constitutional” framework. Prioritizes ethical boundaries and user safety.
• Result: Strong internal safety measures and ethical logic, but external verifiability remains limited.
• Rating: Highest conformance with NIST principles among commercial models, especially in governance and robustness.

3. Google Gemini (1.5)

• Description: Successor to PaLM 2, optimized for integration across Google’s ecosystem.
• Result: Reasonable technical performance, limited insights into model governance or incident response. Explainability still underdeveloped.
• Rating: Incomplete in all NIST categories, but improving with recent research papers.

4. Meta LLaMA (v3)

• Description: Open-source large language model. Trained and published with minimal internal constraints or governance.
• Result: High transparency due to open access. However, lacks formal safeguards, safety evaluations, or fairness assessments.
• Rating: Technically open, ethically minimal.

5. Mistral

• Description: A high-performance French model, open source by design.
• Result: Offers competitive performance, but safety, bias mitigation, and risk handling are entirely absent from documentation.
• Rating: Excellent in transparency, poor in all other NIST criteria.

6. xAI Grok

• Description: Elon Musk’s project intended for Twitter/X integration, with live world knowledge via real-time web access.
• Result: Lacks any form of external governance, explainability, or ethical design. No public documentation on safety or fairness.
• Rating: Technically ambitious, structurally unaccountable.

7. Baidu ERNIE

• Description: A Chinese-language model developed by Baidu, optimized for domestic use.
• Result: Government-aligned, lacks transparency and contains censorship mechanisms. No public evaluation or accountability.
• Rating: Fails on almost every aspect of the NIST framework.

8. Cohere Command R+

• Description: A retrieval-augmented generation (RAG) model focused on enterprise deployments.
• Result: Prioritizes privacy and security in user interaction. Less public information on ethical or governance frameworks.
• Rating: Moderate in safety, strong in privacy, weak in fairness and governance.

---

Comparative Analysis

Model	Governance	Risk Mapping	Evaluation	Management	Explainability	Privacy	Fairness	Transparency
GPT-4	⚠️	⚠️	❌	⚠️	❌	⚠️	⚠️	⚠️
Claude 3	✅	⚠️	⚠️	⚠️	⚠️	✅	⚠️	⚠️
Gemini 1.5	⚠️	⚠️	⚠️	⚠️	⚠️	⚠️	⚠️	⚠️
LLaMA 3	❌	❌	❌	❌	❌	⚠️	❌	✅
Mistral	❌	❌	❌	❌	❌	⚠️	❌	✅
Grok	❌	❌	❌	❌	❌	❌	❌	⚠️
ERNIE	❌	❌	❌	❌	❌	❌	❌	❌
Command R+	⚠️	⚠️	⚠️	⚠️	⚠️	✅	❓	✅

Key Insights:

• Claude 3 leads in ethical alignment and safety architecture.
• GPT-4 remains operationally strong but opaque.
• Open-source models (LLaMA, Mistral) provide transparency but lack reliability safeguards.
• Grok and ERNIE illustrate the dangers of unregulated or state-censored AI.
• Cohere’s model is a promising mid-tier option for secure deployments.

---

Recommendations for Improvement

1. Independent Governance

AI developers must establish third-party oversight mechanisms, possibly via academic consortia or government certification.

2. Formal Risk Mapping

AI models should include published context-of-use diagrams and explicit in/out-of-scope domains to guide safe deployment.

3. Explainability by Design

All models must develop interpretable output tracing, including influence tracking, confidence reporting, and decision trees.

4. Global Transparency Standards

A global “Model Card Standard” should be adopted, much like nutrition labels, incorporating all NIST risk dimensions.

5. Incident Disclosure Mandate

Providers should be required to publish safety incidents, model regressions, and adversarial findings in a public log.

---

Bibliography and Literature Review

Frameworks & Standards

• NIST. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 600-1.
• OECD (2022). Principles on Artificial Intelligence.
• EU Commission (2024). EU AI Act Final Text.

Primary Model Papers

• OpenAI (2023). GPT-4 Technical Report.
• Anthropic (2024). Constitutional AI: Harmlessness from AI Feedback.
• Google DeepMind (2024). Gemini 1.5 Technical Overview.
• Meta AI (2024). LLaMA 3 Release Notes.
• Mistral AI (2024). Introducing Mistral-7B.
• Cohere (2024). Command R+: A RAG-Centric Language Model.
• xAI (2024). What is Grok?.
• Baidu (2023). ERNIE Bot Launch Statement.

Critical Perspectives

• Binns, R. (2018). Fairness in Machine Learning: Lessons from Political Philosophy.
• Selbst, A. D. et al. (2019). Fairness and Abstraction in Sociotechnical Systems.
• Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Machine Learning.

---

Closing Note

The NIST AI RMF offers a rigorous and adaptable structure for assessing AI system risk. As foundation models proliferate, our collective responsibility is not just to develop more powerful tools—but to do so responsibly, visibly, and verifiably. Only through structured transparency and collaborative governance can we ensure that artificial intelligence serves the public good.

---

Would you like this in PDF, with charts or visuals, or submitted as a whitepaper format?

Pdf NIST-rapport