Kubernetes Migration & Event-Driven Scaling for a Browser Agent Fleet

Overview

Browser automation at scale is deceptively hard. Each agent carries live session state and per-agent configuration that make the workload feel stateful — but it is actually stateless once you design around it correctly. This client had reached a point where their StatefulSet-based agent fleet was a bottleneck: scaling up required manual intervention, and scaling down risked evicting agents mid-session.

The engagement covered two parallel tracks: rearchitecting the workload topology, and fixing a shipping problem in the Bitbucket Pipelines CI/CD setup that had been causing stale image tags to reach production.

The Problem

The agent fleet was deployed as a Kubernetes StatefulSet — the standard choice when pods need stable network identities or persistent local storage. In this case, neither requirement actually applied once the architecture was re-examined. The stateful framing had been inherited from an early prototype and never revisited.

Consequences:

• No horizontal auto-scaling: HPAs cannot target StatefulSets with rolling pod management effectively; KEDA support for StatefulSets is limited.
• Slow scale-out: new pods had to be brought up sequentially, not in parallel.
• Eviction risk: when scaling down, Kubernetes has no built-in way to protect a pod that is actively processing a job.
• CI confusion: a latest-tag pinning bug in the pipeline meant Kubernetes was sometimes pulling a cached older image rather than the newly built one, causing silent production mismatches.

The Solution

StatefulSet → Deployment Migration

The migration required verifying that no pod identity assumptions existed in the application code — no $(POD_NAME) hostnames, no persistent volume claims per pod, no ordinal-indexed configuration. Once confirmed, the workload was converted to a standard Deployment.

This unlocked:

• Parallel scale-out: new replicas come up simultaneously rather than one by one.
• KEDA compatibility: KEDA's ScaledObject can target Deployments natively across all scaler types.
• Normal rolling updates: RollingUpdate strategy with proper maxSurge/maxUnavailable tuning.

KEDA Event-Driven Auto-Scaling

KEDA (Kubernetes Event-Driven Autoscaler) was introduced to drive replica count from queue depth rather than CPU/memory averages — a much more accurate signal for a job-processing workload.

# ScaledObject targeting the agent Deployment
spec:
  scaleTargetRef:
    name: agent-worker
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
    - type: rabbitmq
      metadata:
        queueName: agent-jobs
        queueLength: "5"

At idle the fleet holds a minimum warm pool; as the queue grows, KEDA scales out to cap latency; as it drains, it scales back in.

Pod Deletion Cost — Protecting Active Agents

The remaining problem was controlled scale-in. When KEDA reduces replicas, Kubernetes picks pods to terminate based on various heuristics. An agent mid-session should not be the one to go.

The solution uses the controller.kubernetes.io/pod-deletion-cost annotation, which the agent process writes to itself via the Downward API and a small patch call to the Kubernetes API server:

# agent sets a high deletion cost when actively processing
patch_pod_annotation(
    pod_name=os.environ["POD_NAME"],
    annotation={"controller.kubernetes.io/pod-deletion-cost": "1000"},
)

# and resets it when idle / between jobs
patch_pod_annotation(
    pod_name=os.environ["POD_NAME"],
    annotation={"controller.kubernetes.io/pod-deletion-cost": "0"},
)

Kubernetes prefers to terminate lower-cost pods, so idle agents get evicted first and active sessions are left to finish cleanly.

CI/CD Pipeline Fix — Deterministic Image Tags

The Bitbucket Pipelines config was tagging every build image as latest in addition to the commit SHA tag, and the Kubernetes manifests were referencing latest. The result: kubectl rollout would succeed but pods would sometimes pull a cached latest from the node, silently running old code.

Fix: remove the latest tag from all manifests and deploy scripts; pin every deployment to the explicit commit SHA tag built in that pipeline run.

# before — unreliable
image: registry.example.com/agent-worker:latest

# after — deterministic
image: registry.example.com/agent-worker:${BITBUCKET_COMMIT}

This alone eliminated a class of "why is my fix not working in prod" incidents.

Observability and Day-2 Operations

Scaling behaviour you can't see is scaling behaviour you can't trust. The platform runs a full observability stack across a dual-cluster EKS setup (UAT and production):

• Prometheus scraping cluster, KEDA and application metrics — queue depth, replica counts and per-agent job timings on shared dashboards
• Grafana dashboards tracking the scale-out/scale-in cycle against queue pressure, so capacity tuning is a data decision
• Loki for centralised logs across the agent fleet — one query instead of kubectl logs archaeology across twenty pods
• Alertmanager routing that pages on symptoms (queue latency breaching SLO) rather than causes (a single pod restart)

The same stack underpins incident response across the wider platform — the difference between "users are reporting failures" and "queue latency crossed threshold four minutes ago, here's the deploy that did it."

• Kubernetes
• KEDA
• AWS EKS
• Bitbucket Pipelines
• Prometheus / Grafana
• Loki / Alertmanager
• RabbitMQ
• Python

Outcome

• Agent fleet scales automatically from a minimum warm pool to 20 replicas based on queue depth, with no manual intervention required.
• Pod deletion cost scheduling prevents active browser sessions from being terminated mid-job during scale-in events.
• The CI/CD pipeline now ships a deterministic image reference on every deploy, eliminating silent image staleness.
• Rolling deploys to the agent fleet are zero-downtime and take roughly 90 seconds end-to-end.
• Queue-depth, scaling and latency metrics live on shared Grafana dashboards — capacity decisions are made from data, and incidents are caught by alerts rather than user reports.