Would You Trust This AI Risk Assessment?

DeMeakey Williams Sr. — Founder & CEO of Global AI Risk® / Crane Consulting and Technology Solutions — will present Would You Trust This AI Risk Assessment? at Mond(ai)y Coffee on Monday, October 5.

What are you building?

We’re building Global AI Risk® (GAIR), an AI risk intelligence and governance platform designed to help organizations understand the risk associated with AI systems before they deploy them and as those systems change over time.

We’re initially focused on regulated and operational environments such as airports, government, and critical infrastructure.

The core idea

AI risk intelligence should be based on evidence, not assumption.

GAIR evaluates AI systems across regulatory compliance, cybersecurity, ethical bias, and data privacy, while also measuring transparency. More importantly, we show the evidence behind the assessment so the operator doesn’t simply have to trust a number.

What’s working / not working

• Working: GAIR can produce an assessment, break it into individual risk areas, identify the documents and evidence supporting those findings, surface adverse findings, and recalculate the assessment as new evidence becomes available.
• Not working (yet): We’re still stabilizing and validating the scoring methodology. My CTO is focused on making the scoring reasonable, repeatable, stable, and defensible. Meanwhile, I’m already in enterprise business-development conversations, so I need to understand when the evidence and validation are sufficient for a customer to reasonably rely on the assessment.

A real example

GAIR recently assessed a major enterprise software vendor’s role-specific AI agents:

Overall assessment: 73/100 — Moderate Risk

• Regulatory Compliance — 78
• Cybersecurity — 78
• Ethical Bias — 55.6
• Data Privacy — 78
• Transparency — 85

The assessment used 8 published sources, and every source was opened during the assessment. The operator can see the supporting documents, governance policies, certifications, transparency reports, adverse findings, and source links behind the assessment.

So my problem isn’t simply, “Can we show evidence?”

It’s:

“When is that evidence sufficient to justify the conclusion we’re presenting?”

Open questions

• What would you need to see behind this 73/100 assessment before you would trust it enough to use it in an actual AI deployment decision?
• When there isn’t an obvious ground-truth “correct” AI risk score, how should we validate that the conclusion we’ve drawn from the evidence is defensible?
• If two AI systems receive the same score, but one is supported by extensive independent evidence and the other by limited vendor disclosures, should an executive have the same confidence in both? How should we communicate that difference?

Spicy takes

Showing your evidence may ultimately be more important than showing your score.

A numerical score simplifies the decision, but the real value may be giving an executive enough evidence, traceability, and context to understand why the assessment exists and how much confidence to place in it.

If we only discuss one thing, it should be:

I’ll put the actual GAIR assessment and its supporting evidence on the screen and ask:

You can see our score. You can see our evidence. What else would you need before you would trust this assessment enough to use it in an actual AI deployment decision?