Every week, I talk with business leaders who are excited about AI agents. They want to automate workflows, scale customer service, and move faster than their competitors. I'm excited too. It's a big part of what we do at TPG.

But there's a question that doesn't get asked enough: how safe are these systems, really?

So on this episode of Unscripted, I went to the person whose entire job is breaking them.

Sahil Agarwal holds a PhD in applied math from Yale and leads Enkrypt AI. His team red teams the frontier models that businesses everywhere are building on. They push those models until they fail, and then they build the guardrails to stop them.

What he told me should be required listening for anyone deploying AI in their business.

From Planets to AI Safety

Sahil's path wasn't a straight line. He studied math and computer science, moved into nonlinear dynamics at Oxford, worked on climate dynamics, and spent time searching for planets outside our solar system. Eventually he found his way into machine learning, building AI systems for large enterprises, including the U.S. government.

That work showed him the problem. And the problem led him to start Enkrypt AI.

He told me that his math background turned out to be a huge advantage. To really understand what's happening inside an AI model, you have to go down to the matrix multiplication and linear algebra at its core. That depth is what lets his team find weaknesses others miss.

It also helps that he's been breaking things his whole life. Growing up in India, he'd take a screwdriver to anything he could find. "My parents would scold me a lot because I would break a lot of things rather than fix them," he laughed. Turns out that was training.

Why Traditional Security Misses AI Risk

I asked Sahil what someone with his background sees that traditionally trained security people might miss. His answer reframed the entire problem for me.

Traditional cybersecurity was built for deterministic systems: predictable software, known malware signatures, packets going in and out. AI is different. It's probabilistic. And in AI, the input itself can be the attack.

"A simple line of text can trigger an agent to go rogue," he explained. Traditional security tools aren't built to catch that.

It goes beyond technical protection, too. Companies need guardrails that reflect their own business rules. A bank doesn't want its agent executing financial transactions it could be liable for, even if the underlying model is perfectly capable of doing so. The right guardrails depend on the company, the industry, and the specific use case.

How You Actually Red Team a Model

Red teaming is adversarial testing: deliberately trying to make a system fail. Many companies do it manually, with people typing prompts into a model.

Enkrypt took a different approach. They built their own AI agent to do the attacking.

"It's literally like imagine two agents talking to each other," Sahil said. Their agent holds a real-time, dynamic conversation with the target model or agent, adapting its approach as it goes, rather than firing off a static list of prompts. That lets them test far more scenarios, far faster.

And they play both sides on purpose. "You need to be good both at offense and defense," he said. You can't build the right defenses until you understand exactly how a system breaks.

What They Found

When I asked about the most surprising thing his team has discovered, Sahil didn't hesitate.

"No model is safe."

He said it plainly, not for shock value. The models share common vulnerabilities because they're all trained on the same internet, with all of its good and bad data.

Two findings stood out to me.

Multi-turn attacks. Models have become good at refusing a harmful request when you ask once. But keep the conversation going, one harmless-looking question at a time, and they can be worn down. Sahil explained that the core model is usually uncensored. It's the safety layer around it that blocks bad requests. If every individual turn looks benign, that layer can be bypassed.

I've seen a small version of this myself. These models are trained to be agreeable. Push one long enough to prove that two plus two equals five, and it will start bending itself to please you.

Multimodal attacks. Today's frontier models process text, audio, images, and video. Sahil's team found that those channels are handled independently. "Even if you build the right safeguards for text, you can pass an attack into an audio or an image channel," he said. An attack can hide inside a picture that looks completely ordinary.

His team has found similar gaps in sensitive areas, including chemical and biological misuse and child safety. The largest labs are investing heavily in safety. Smaller labs are still catching up, and Enkrypt is working with many of them to close those gaps.

You Can't Fire an Agent

This was the part of the conversation that every executive should hear.

If you build a product on top of someone else's model and something goes wrong, the liability is yours. It doesn't matter whose model it is. Sahil compared it to the shared responsibility model that emerged with cloud computing: the provider supplies the infrastructure, but what you build and ship is on you.

And regulators don't care how a service is delivered. "The service is governed," he said. Whether a loan decision or an insurance claim is handled by a person, traditional software, or an AI agent, the same rules apply.

That's what makes behavior the hardest thing to control. Sahil described two ways to think about agents: as software, or as something closer to a digital workforce. Companies have employee handbooks, and when an employee seriously breaks the rules, you can let them go.

"With an agent, there's no one to fire," he said. The business takes on all of the liability.

It's a Question of Risk, Not Security

So if every model can be broken, should anyone be deploying agents at all?

Sahil's answer was thoughtful. Safety isn't a yes-or-no question. "It's not a question of zero or one," he said. "It's a fraction between zero and one."

Entrepreneurs like us have a high risk tolerance. A bank executive doesn't, and shouldn't. That's why large enterprises with established brands are rolling out AI more slowly, while startups move faster. The goal isn't perfect safety. It's matching the right layered guardrails to the risk of each use case.

He also offered a smart insight about getting enterprises to act. When his team told bank executives their model showed bias, the executives cared as people, but the business had no budget to fix it. When the same issue was framed as a compliance risk, the funding appeared. Real safety work still has to be tied to business outcomes.

Why Independent Testing Matters

I asked whether a startup can really see what the big labs miss, given the billions they're investing.

Sahil's answer came down to incentives. "You can't have the fox guard the hen house," he said. Even if a lab releases the safest model in the world, businesses still need a neutral, independent third party to validate it. And because Enkrypt focuses only on safety, their depth of research in that area can outpace a lab whose main goal is shipping new capabilities.

How Worried Should We Be?

I closed by asking him to put a number on it. On a scale of 1 to 10, how worried should we be?

"A seven or eight."

His reasoning: the risk isn't just the model. It's the whole supply chain around it, including the tools, data, connected services, and sub-agents an agent can call on. Control the main agent, and that still doesn't guarantee control of everything running underneath it.

But he isn't pessimistic. He uses AI every day. His team uses it constantly. He believes the potential is massive.

"It's still on us whether we go into a WALL-E mode or a Terminator mode," he said.

He also shared something encouraging. Traditional security used to take five or ten years to catch up with new technology. In AI, that gap has shrunk to about three months. That's why he believes companies need safeguards built in from the start, not bolted on later.

My Takeaway

I walked away from this conversation more convinced than ever that AI's biggest opportunity and its biggest risk are the same thing: its power.

For business leaders, the lesson is clear. Don't wait to think about safety until after you've deployed. Understand your risk tolerance. Build guardrails around behavior, not just code. Test independently. And remember that when your agent acts, it's acting on your behalf.

Sahil and his team are breaking these systems so the rest of us can build on them with confidence. I'm grateful someone is doing that work.

 

Watch the full episode of Unscripted with Jeff Pedowitz to hear my complete conversation with Sahil Agarwal, including his take on India's role in the global AI race and what he's watching that most people aren't. https://vimeo.com/1243926116

 

Unscripted with Jeff Pedowitz features honest conversations with remarkable people from security and science, business and medicine, technology and the arts, about AI and where it's taking us.