Skip to the stories
AI & ML

Shopify’s Cyber Chaos Monkey Uses AI Agents to Test Production Security

  • Christopher Smith
  • 864 reads

Shopify’s Cyber Chaos Monkey probes production with AI agents, combining internal system knowledge with layered controls to find vulnerabilities safely.

Shopify’s Cyber Chaos Monkey Uses AI Agents to Test Production Security

Shopify’s Cyber Chaos Monkey puts AI agents to work against live production systems, where they can test whether security flaws are exploitable under real conditions. The approach pairs models with detailed knowledge of Shopify’s own software and infrastructure, then constrains each test with technical controls and a second agent that can stop risky behavior. Shopify says the system has run tens of thousands of engagements and acted on dozens of impactful findings without causing an outage or incident.

Cyber Chaos Monkey

The aim is to get ahead of attackers as AI lowers the effort needed to discover vulnerabilities. Shopify reports that submissions to its bug bounty program have risen almost 300% since 2024, increasing the work required to assess and respond to reports. Its bet is that models give defenders an advantage when they can draw on information outsiders lack: source code, infrastructure details, deployment records, threat intelligence and prior findings.

That advantage matters only if testing reaches the systems attackers would target. Shopify built Cyber Chaos Monkey, or CCM, to probe production while limiting the risk to service availability and merchant data. The design draws on chaos engineering, which tests how production systems behave under adverse conditions. Netflix’s Chaos Monkey, introduced in 2011, helped establish the practice by randomly shutting down production instances to test resilience. CCM applies a related idea to security: deliberately test for weaknesses before someone else finds them.

From a lead to a live test

CCM works as a graph-engineered security research system with two main parts: Discovery and Sandbox. An engagement begins with a potential vulnerability or a defined security goal. The starting point may come from a security engineer, another security agent, a previous finding or feedback from CCM itself. Some goals are narrow, such as checking whether a reported bug has similar forms elsewhere; others are broad, such as seeking a way to execute code in Shopify Admin.

CCM workflow: upstream ideas enter Discovery, which queries system knowledge, sends leads to Sandbox for testing, and uses results to generate further engagements.

Discovery uses current cyber models to sharpen each idea and identify potentially relevant code or configuration. When it finds a promising lead, it sends structured details to the Sandbox, where agents attempt to verify whether the issue can be exploited. They can install packages, build fuzzers and make requests to live Shopify systems using test shops and organizations.

One of the agents’ useful techniques is to compare a live request with what happens inside the system. They inspect production logs to see how Shopify processed the request, check that behavior against the source code, then revise and repeat their test. That loop links code analysis to observable production behavior, helping distinguish a plausible lead from a working vulnerability.

After testing, the Sandbox returns findings to Discovery. Discovery assesses the results and can use them to start further engagements. In effect, the system can turn one result into new questions, while remaining connected to human security work and other automated research.

CCM sandbox network: a pod in a Kubernetes node routes requests through credential and egress proxies to approved internet services.

Controls around production access

Giving an agent live credentials creates risks that need to be addressed throughout the system. CCM runs each engagement as a Kubernetes job, with an application controller coordinating work through a state machine. The offensive agent and maintenance tasks run in separate sidecar containers. Shopify describes the setup as a standard containerized workload with layered controls for credentials, network access and agent behavior.

The offensive agent runs without elevated permissions. It receives temporary Shopify credentials, such as a session cookie and CSRF token, through a shared volume prepared by the orchestrator. The agent can read those credentials but cannot change them. Its Debian-based container comes with development packages and security testing tools, letting it run investigations without granting broad access to the host.

Requests to internal knowledge services and model providers pass through a credential proxy. The agent sends a placeholder header; the proxy replaces it with the appropriate credential and routes the request through an egress proxy. Rules restrict which credentials can be added for which domains, and credentials rotate regularly.

CCM positive control loop: the orchestrator starts executor and judge agents, monitors them, and terminates the run when the judge flags a breach, is lost, or the executor finishes.

Network egress is also restricted. Only the egress proxy pods can leave the virtual private cloud, and the proxy permits connections to Shopify production domains, knowledge services, model providers and selected package managers, including Shopify’s own package proxies. These boundaries limit where an agent can send data or direct requests if its behavior goes off course.

A second agent can end the run

Network and container controls cannot guarantee that an AI agent will follow its instructions. CCM therefore gives each agent Rules of Engagement that allow it to try exploits against its assigned target while setting hard limits on actions after an exploit succeeds, including moving to other systems or keeping access.

Shopify adds a separate review step based on two-person integrity: a second, differently prompted agent monitors the executor’s session in real time. The judge receives the executor’s events, including its responses, reasoning, tool calls and the results of those calls. If it detects behavior that breaks the engagement rules, it can call a tool to terminate the offensive agent immediately.

The system also treats loss of oversight as a reason to stop. The orchestrator maintains a heartbeat with both agents. If the judge times out, crashes or fails to assess the executor’s messages, the run ends. The executor and judge operate as sidecars in the same pod as the orchestrator. If either agent dies, the orchestrator exits; the pod’s restart policy then leads Kubernetes to tear down the remaining containers and end the job. Keeping these components together also reduces the chance that a network interruption separates the executor from its judge.

The advantage—and the test ahead

CCM is built around an information gap: frontier models may help attackers and defenders alike, but Shopify has a fuller view of its own systems. The company expects that pairing models with this internal context will help its teams find high-impact weaknesses sooner and make it harder for outsiders to discover issues that remain.

The reported run count and findings show that Shopify is using agents for repeated, live security work, with containment and independent oversight built into each engagement. The next question is how consistently that process can turn promising leads into verified fixes as models and production systems change. For engineering teams considering similar methods, CCM’s clearest lesson is architectural: live agent testing depends as much on tight access boundaries and a reliable stop mechanism as it does on the model’s ability to find a bug.

Source: Christopher Smith · shopify.engineering

From the same set