Four shapes, one cluster, very different operational headaches.

“Agent” is doing some heroic heavy lifting as a word right now.

Depending on who you ask, an agent might be:

  • A python script firing off 10,000 parallel sub-second rollouts for reinforcement learning.
  • An ephemeral sandbox executing untrusted code generated by a chatbot.
  • Headless Claude Code hacking on a repo for two hours with a browser and a terminal.
  • An always-on personal assistant that reads your emails and chats you questions about how to respond.

Each of these use cases is an “agent” but you can’t just run them like a standard Kubernetes microservice.

When I look at agent architectures (especially on Kubernetes), how you run them comes down to a fundamental tradeoff: churn versus density.

The agent workload spectrum: from high-churn swarms to hibernating density.

On the left, you scale by tearing through ephemeral environments as fast as your control plane can spin them up. On the right, you scale by packing long-lived, mostly-sleeping agents onto nodes without running out of memory.

To handle this spectrum without turning your cluster into a dumpster fire, GKE Agent Sandbox introduces native Kubernetes Custom Resource Definitions (CRDs)—like Sandbox and SandboxWarmPool—giving you first-class APIs to manage agent lifecycles, security boundaries, and idle states.

Let’s break down the four distinct shapes across that line, what breaks when you run them naively, and how to make Kubernetes handle them cleanly.

1. The Mayfly: Reinforcement Learning & Eval Swarms

Born to run an eval rollout, lived for four seconds, died for the gradient update.

In nature, mayflies hatch by the millions, do their thing in a frantic swarm, and vanish within hours.

In AI workloads, this is your distributed RL training loop (think GRPO or PPO) or large-scale benchmark harness. The model generates trajectories across thousands of parallel environments, computes advantages, updates weights, and tosses the environments in the trash.

What breaks

Your bottleneck here is pure lifecycle churn.

If you ask a standard Kubernetes cluster to spin up and tear down 20,000 pods an hour, etcd and the kubelet will probably stage a walkout. The API server gets hammered with object creation events, node status updates queue up, IP address allocations thrash the CNI plugin, and pod cleanup lag leaves ghost resources scattered everywhere.

The cluster fix

Don’t let these sandboxes linger. Define lightweight Sandbox resources with aggressive teardown policies so completed evaluation runs vanish without piling up orphaned objects in etcd:

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: rl-rollout-worker-4821
spec:
  shutdownPolicy: Delete          # Throw away the sandbox immediately on exit
  podTemplate:
    spec:
      restartPolicy: Never
      containers:
        - name: env
          image: rl-environment:latest
          resources:
            requests: {cpu: "500m", memory: "512Mi"}
            limits: {cpu: "500m", memory: "512Mi"}

If you are running millions of these, you quickly realize that spinning up full Kubernetes pods for a four-second Python evaluation is too much overhead. You want batch frameworks (like Ray on GKE or custom worker pools) that keep worker pods alive and dispatch ephemeral tasks inside them. But when you do manage rollouts as discrete sandboxes, setting shutdownPolicy: Delete is mandatory if you want your control plane to survive the day.

2. The Hummingbird: Ephemeral Code Execution

Fast, twitchy, and executing arbitrary user Python straight from the prompt.

Hummingbirds move at blistering speeds and burn through energy at a dizzying rate.

This is the short-lived tool sandbox. A user asks an LLM to analyze a CSV, the model writes a quick Python script, and your backend executes it to return a chart. Lifetime: 2 to 10 seconds.

Unlike the Mayfly, we don’t really know much about what this code is (ie, it’s not just the same test suite you run for your RL rollouts). This means we can’t optimize for that one task. Instead, we have to assume the worst and place it in a sandbox.

What breaks

You are executing arbitrary code written by an LLM on behalf of external users. You cannot run this directly on your host kernel with standard container isolation. You need a hardened sandbox boundary like GKE Sandbox (powered by gVisor).

Here’s the hard part: spinning up a hardened sandbox from a cold start takes time. Pulling images, configuring the runtime, and booting the sandboxed kernel takes 1–3 seconds. If your user is waiting on a chat response, adding a three-second sandbox boot penalty on top of model inference latency destroys the interactive feel.

The cluster fix

Pre-warm your sandboxes.

Instead of creating a new pod on demand, you keep a warm pool of pre-initialized, isolated sandboxes sitting in memory ready to claim instantly:

apiVersion: agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
  name: python-interpreter-pool
spec:
  replicas: 10                    # Pre-booted and waiting
  sandboxTemplateRef:
    name: interpreter-template

When a tool call comes in, you grab an already-booted sandbox from the pool in milliseconds, run the code, return the stdout, and destroy or recycle the environment. The user gets sub-second execution without sacrificing the kernel boundary.

3. The Octopus: Long-Lived Coding & Browser Agents

Eight tools, a checked-out git branch, and a lot of time spent waiting on LLM calls.

An octopus agent is a jack of all trades (and somehow master of all too???). It manipulates multiple tools simultaneously, and stays anchored to its home (directory).

This is your autonomous coding agent or browser automation session (OpenClaw, Claude Code, Antigravity, etc.). It runs for anywhere from twenty minutes to three hours. It clones a git repo, edits files, runs tests, reads documentation in headless Chrome, and iterates.

These agents spend most of their lives in intermittent bursts. They fire off a prompt to an LLM, wait 15 seconds for tokens to stream back, run a build, wait for a test suite, and then sit idle for ten minutes while you go grab a coffee, or for 48 hours while you enjoy the weekend away from your desk.

What breaks

Two classic headaches hit you here:

  1. State loss: If the pod restarts or gets rescheduled during a node drain, you lose the git workspace, shell history, and local state.
  2. Re-connection: You need a reliable way to reconnect your IDE or browser UI to the exact same running session.

The cluster fix

Treat the agent like a stateful workspace with a dedicated persistent volume and a stable DNS endpoint, backed by a retain policy:

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: coding-agent-auth-refactor
spec:
  service: true                 # Stable DNS endpoint to reattach
  volumeClaimTemplates:
    - metadata:
        name: workspace
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests: {storage: 20Gi}
  podTemplate:
    spec:
      runtimeClassName: gvisor
      containers:
        - name: agent
          image: coding-agent:latest
          volumeMounts:
            - name: workspace
              mountPath: /workspace
  shutdownPolicy: Retain        # Keep the disk and work safe when the pod stops

It’s the best of all worlds when you can schedule a whole team’s coding agents on a single cluster and let them share raw resources like compute without having to share an environment (permissions, tokens, etc.). I’m planning to show how to do that on GKE soon!

4. The Bear: Persistent Personal Assistants

Hibernating 95% of the day. Still holding OAuth tokens. The cloud bill is watching.

Bears spend a huge chunk of the year hibernating, but when they’re awake, they hold serious authority in the woods.

This is the dedicated personal assistant. It lives indefinitely, maintains user-specific memory, holds delegated OAuth tokens, and acts on your behalf.

The operational dilemma here is straightforward: it is idle 95%+ of the day, but when it wakes up, it needs access to real tools and credentials.

What breaks

  1. The Idle Tax: If you run 5,000 personal assistant pods for everyone in your company that each reserve 2GB of RAM, you are paying for 10TB of memory just to host agents that are literally doing nothing while users sleep.
  2. Blast Radius: A long-lived agent with accumulated credentials is a high-value target. Giving it a static admin service account key is asking for trouble.

The cluster fix

Put the bear to sleep.

Instead of keeping the pod running 24/7, you suspend the sandbox when it’s inactive:

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: assistant-drew
spec:
  operatingMode: Suspended      # Releases CPU and RAM while idle
  service: true
  volumeClaimTemplates:
    - metadata:
        name: memory
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests: {storage: 10Gi}
  podTemplate:
    spec:
      containers:
        - name: assistant
          image: assistant:latest
          volumeMounts:
            - name: memory
              mountPath: /var/lib/assistant

When an event or message arrives, wake it back up:

kubectl patch sandbox assistant-drew --type=merge \
  -p '{"spec":{"operatingMode":"Running"}}'

When suspended, the pod releases its CPU and RAM allocations. The persistent volume, network identity, and configuration remain pinned. You can park thousands of suspended assistants on a handful of nodes because you are only paying for disk storage instead of active compute. And you can manage credentials outside of each pod so that the sandboxed agent doesn’t hold on to long-lived credentials while it’s sleeping.

What’s Coming Next

What makes this space fun is watching Kubernetes adapt to these patterns. The core primitives in GKE Agent Sandbox are landing right now: isolated gVisor sandboxes, pre-warmed pools to eliminate startup tax, durable volume retention, and native suspend/resume mechanics.

Beyond these initial CRDs, there’s even more frontier tech coming down the pike — like Agent Substrate. As agentic workflows evolve from single assistants into complex multi-agent graphs and massive fleets, Agent Substrate will tackle the deeper platform-level orchestration, dynamic environment routing, and cross-agent scaling challenges that arise when you run thousands of cooperating agents in production.

What I’m Experimenting With

Helpful Resources & Hands-on Tutorials

Which of these beasts is your agent? Are you dealing with high-churn evaluation swarms, or trying to figure out how to host a fleet of long-lived coding agents without blowing up your memory budget? I’d love to hear what your clusters look like!