Can an insurer actually price the risk of an AI system before it fails in the real world? According to Rajiv Dattani, who helped run Meter, the organization that tested whether OpenAI's and Anthropic's frontier models were safe to release, the answer is yes, and it works differently than almost any other line of insurance. Instead of waiting years to accumulate real loss data, AI risk can be simulated upfront through red-team style evaluations that estimate how a system will fail, how often, and how severely, before it ever reaches production. That capability is the foundation of the company Dattani left frontier AI safety evaluation to build: insurance for AI, backed by Lloyd's of London.

This episode of Unscripted covers what it actually meant to sit in the chair deciding whether a model was too dangerous to release, why he thinks the viral stat about 95% of AI pilots failing is good news rather than a warning, and what history's insurers can teach today's AI industry about moving faster than regulation.

Who is Rajiv Dattani?

Rajiv Dattani is an Oxford-trained economist and former McKinsey insurance partner who helped run Meter, the organization responsible for evaluating whether frontier AI models from labs like OpenAI and Anthropic were safe enough to release to the public. He has sat on both sides of the AI risk question: assessing whether a system is safe before launch, and now building the financial infrastructure that determines who pays when a deployed AI system fails. His current work centers on a standard called AUC-1, built with a consortium of roughly 250 security and risk leaders, and underwritten through Lloyd's of London, the world's oldest insurance marketplace.

What did Meter actually test before a frontier model could be released?

At Meter, Dattani's team measured a model's capability to conduct AI research and development tasks, including a specific concern: how capable a model is at contributing to its own successor's training. The output of these evaluations was a score expressed in human hours, meaning how long it would have taken a person to complete the most complex task the AI could accomplish. As models complete tasks that would take a human 10, 20, or 40 hours, Dattani explains, the risk increases that AI systems could begin training future AI systems with less human involvement, pushing the industry closer to a threshold Meter's reports were designed to track.

Why leave AI safety evaluation to build AI insurance?

Dattani's reasoning draws on a pattern he sees repeating across technology history: markets tend to identify and price risk faster than regulators can catch up, and insurers in particular have both the incentive and the resources to lead that work. He points to Benjamin Franklin's mutual fire insurance company in Philadelphia, which audited home construction materials and tree spacing to determine fire risk long before those practices became formal building codes. He also cites Progressive, which offered a 20% discount on auto insurance premiums to drivers with seatbelts and airbags in the 1950s, well before either was required by federal law. His company is built on the same logic: use market incentives to surface the safety measures that matter before legislation catches up.

How do you price AI risk without years of historical loss data?

Dattani's company does not act as the insurer directly. It works with insurers at Lloyd's of London, providing the underlying risk assessment that insurers use to set terms and back policies with their own balance sheets. Traditional insurance lines require accumulating real claims data over time, the way early auto insurers underwriting Uber initially lost money before they had enough data to price the risk profitably. AI insurance is different because evaluations and red-teaming can simulate how a system will perform and fail before it is ever deployed, generating a synthetic loss distribution instead of waiting for real-world losses to accumulate. Dattani is direct about the limits of relying on old data: pricing a current AI agent using data from the first release of ChatGPT would be so far out of distribution that it would be actively unhelpful.

Is 95% of AI pilots failing actually a bad sign?

No, according to Dattani, and he pushes back directly on the MIT study that went viral for reporting a 95% AI pilot failure rate. His view is that pilots should be treated as experiments, meaning they are supposed to fail often. A high failure rate means companies are killing the ideas that don't work early and can focus resources on the 5% that do. His concern is the opposite reaction: if business leaders expect most pilots to succeed, they will avoid taking the ambitious swings that produce real breakthroughs.

Why do most AI pilots fail for management reasons, not model reasons?

Dattani argues that AI models don't operate like a one-to-one swap for a human role. Instead, leaders need to break down which specific tasks within a role an AI model can support or outperform, then redesign how the organization operates around that division of labor, with the right human oversight layered in. He describes this as a deep management and organizational design challenge, not a model capability problem, and notes that many managers don't understand the granular processes and data flowing through their own teams well enough to know where AI actually belongs.

What is AUC-1, and how does it decide if a system is safe to deploy?

AUC-1 is the risk standard Dattani's company built to enumerate every known category of AI failure and the corresponding preventative measures. It draws on three sources: real-world incidents already reported in the press, academic research on model failure modes that haven't yet shown up publicly, and qualitative input from security and risk leaders across enterprises and AI labs. Each risk in the taxonomy, such as data leakage from improperly filtered PII or an AI agent gaining access to systems it shouldn't, is mapped to specific preventative measures. A system only passes AUC-1 once it has both passed simulated evaluations and demonstrated the safeguards are actually in place, across categories including security risks like jailbreaks and prompt injection, safety risks like harmful outputs, and reliability risks like hallucinations and faulty tool calls.

Can an AI system just be uninsurable?

Yes, and Dattani says that's acceptable as long as it's transparent. Some risks, like systems touching sensitive health data or critical infrastructure, require a level of reliability where even a 99% pass rate isn't good enough because the cost of the remaining 1% is too severe. In those cases, he says the more important step, one that mostly hasn't been happening, is simply determining and disclosing how risky a system actually is, so a buyer can knowingly accept the risk rather than have it sit silently, unpriced, on their balance sheet or, at scale, on a government's.

How does an AI insurance standard avoid getting watered down?

Dattani ties his company's financial incentives directly to the rigor of its own standard. Because the company's revenue is connected to the insurers underwriting these risks, watering down evaluations to win more business would directly increase the company's own financial exposure if a breach occurs. He also points to structural transparency: a consortium of roughly 250 security and risk leaders peer reviews every change to the standard, with a public change log documenting what changed and why. He acknowledges this isn't a permanent substitute for eventual federal and international frameworks, but argues the market can lay useful groundwork while formal regulation catches up.

What worries Dattani most about AI risk right now?

Not a single catastrophic failure, but the lack of infrastructure to catch small, compounding errors before they cascade. He draws a direct comparison to Three Mile Island, arguing that one major AI safety incident could trigger a public backlash severe enough to stall progress for a generation, the same way that nuclear incident slowed nuclear energy investment for decades. His specific concern going forward is economic security risk: as AI becomes embedded across banking and payments infrastructure, the industry needs fine-grained incident monitoring capable of catching subtle failures before they become systemic across the economy, an area he believes is currently underfunded relative to the attention going to cyber and biosecurity risks.

FAQ

How does AI insurance price risk without years of historical claims data? AI insurance can rely on simulated red-team evaluations that test how a system fails and how severe those failures would be, generating a synthetic loss distribution before real-world claims exist, unlike most insurance lines that require years of accumulated loss data.

Is a 95% AI pilot failure rate actually bad? According to Rajiv Dattani, no. Treating pilots as experiments means a high failure rate simply reflects healthy experimentation, allowing companies to kill non-working ideas quickly and concentrate resources on the small percentage that succeed.

What is AUC-1? AUC-1 is an AI risk standard that catalogs known failure categories, drawn from real-world incidents, academic research, and input from security leaders, and maps each to specific preventative measures a system must demonstrate before it passes.

Can an AI system be uninsurable? Yes. Some systems, particularly those touching sensitive data or critical infrastructure, may not meet the reliability bar insurers require. Dattani argues this is acceptable as long as the risk level is transparently disclosed to the buyer rather than left unpriced.

Why do AI pilots fail more often for management reasons than technical ones? Because AI models can't simply replace a human role wholesale. Leaders need to identify which specific tasks a model can support or outperform and redesign workflows accordingly, which requires a level of process understanding many managers don't have for their own teams.

Listen to the full conversation with Rajiv Dattani, and catch every episode, on the Unscripted with Jeff Pedowitz page.