The AI Control Loop: When AI Goes Rogue - with Craig Thomas of Wallarm

Episode Summary

Executive Summary: The episode argues that “rogue AI” usually isn’t malicious, but systems acting outside intended boundaries through shadow use, prompt injection, drifting model behavior, and chained agent workflows. Craig Thomas says organizations need end-to-end visibility and runtime enforcement across AI, APIs, and apps to move from detection to true control without slowing innovation.

Main Topics: What “rogue AI” means in practice (Priority: 5/5): Rogue AI is framed as AI that behaves outside intended boundaries, not necessarily maliciously. Craig breaks it into unsanctioned, unmonitored, and unpredictable AI, with agents as the fastest-growing risk because they can take actions on behalf of users. How AI drift happens (Priority: 5/5): AI systems can drift through prompt injection, untrusted data sources, user inputs, or provider model updates that change behavior without code changes. The key issue is that intent may be fine while outcomes become unsafe or noncompliant. Shadow AI and shadow LLMs (Priority: 5/5): Employees or developers may spin up AI tools or LLM endpoints without approval because formal intake is too slow. This creates visibility, logging, and data-handling gaps, and can also introduce over-privileged credentials into agents. Why sequences and chains are the real risk (Priority: 5/5): Single AI actions may look benign, but chained workflows across tools, APIs, and agents can create compliance, privacy, or security violations. End-to-end session tracing is presented as essential to understand the full sequence. APIs as the central nervous system of AI (Priority: 5/5): Every meaningful AI action eventually becomes an API call. Since enterprise environments have large, often undocumented API surfaces, failing to monitor the API layer leaves a major blind spot where exfiltration and abuse materialize. Governance, oversight, and innovation balance (Priority: 4/5): The solution is not to block AI adoption, but to create clear sandboxes, fast approval paths, isolated credentials, scoped access, and runtime monitoring so teams can experiment safely without resorting to shadow behavior. From detection to control (Priority: 5/5): Craig emphasizes that detection alone is insufficient. Real oversight means inspect, block, and rate-limit AI traffic in real time, prove enforcement to auditors, and close the gap between observed risk and prevented damage.

Key Arguments: Rogue AI is usually not malicious; it is more often legitimate systems producing unintended or out-of-bounds outcomes. Prompt injection and untrusted inputs can redirect agent behavior without any direct change to the model or code. Hosted models can drift when providers update them, causing behavior changes without change management or review. Shadow AI grows when official approval processes are too slow, encouraging developers to bypass governance. Chained agent workflows multiply risk because each individual step may pass checks while the overall sequence becomes unsafe. AI must be governed at the session level, tracing prompts, tool calls, model calls, and API responses end-to-end. APIs are the main execution layer for AI actions, so AI security and API security must be integrated. Policy documents alone do not create control; real-time runtime enforcement is required. Organizations need fast, usable sandboxes and fast paths to production to preserve innovation while reducing shadow usage. Detection after the fact does not prevent fines, exfiltration, or harm; control must happen at machine speed.

Data Points: Experience in Department of Energy: 10+ years - Craig Thomas described his background in regulated environments before joining Wallarm. Risk rating of corporate AI tools in active use: 72% high or critical risk - Cited as evidence that AI risk is present today, not just a future concern. Response-time example: 1 minute - Used to show that a one-minute security response can be too slow when an agent may make tens of thousands of API calls. Scale example: 50,000 API calls in a minute - Illustrates how quickly damage can occur before detection or response. Enterprise API surface: Thousands to tens of thousands of endpoints - Highlights how large and often undocumented the API attack surface can be.

Pivotal Quotes: "Policy alone is not oversight." — Craig Thomas: On why governance documents do not stop harmful AI behavior in real time. "Detection without prevention is just documentation of damage." — Craig Thomas: On the difference between seeing AI risk and actually controlling it before harm occurs. "It was working as designed, but not as intended." — Craig Thomas: Used to explain how legitimate AI behavior can still become rogue or harmful.

Implications: Enterprises need unified AI/API governance, session tracing, and inline enforcement to stop risky behavior at machine speed. The winning model is safe experimentation with fast approvals, not blanket restriction or reactive detection.