Chaos Engineering on AWS EKS
Chaos Mesh on EKS, running namespace-scoped pod-kill, latency, and network-partition experiments against a three-tier app on Spot instances, with RBAC pinning every experiment inside its sandbox.
Building this took longer than expected because of three things I got wrong up front. I assumed Docker was the runtime, but Amazon Linux 2023 uses containerd, and Chaos Mesh needs the runtime socket path to match. I initially avoided Spot instances thinking they were too flaky for reliability testing. And I reached for embedded credentials before remembering IRSA existed.
I fixed all three: Helm-installed Chaos Mesh with chaosDaemon.runtime=containerd and socketPath=/run/containerd/containerd.sock, kept Spot instances in the node group because the surprise reclaims are extra chaos for free, and wired IRSA so experiments needing AWS APIs get scoped IAM instead of long-lived keys. A three-tier app (nginx, three httpd replicas, a single Redis) runs in a chaos-sandbox namespace, and curated RBAC plus targetNamespace filtering pins every experiment inside that namespace regardless of what someone types into the dashboard.
The runtime, the node lifecycle, and the credential model all lie to you until you check them.
Experiments now run in a namespace-scoped sandbox on Spot with a limited-scope 2-hour dashboard token, and the progressive scenarios (pod-kill, 500ms latency, 15% packet loss, database partition) each produce reproducible before/after evidence.
- chaosDaemon pinned to containerd socket path for AL2023 nodes
- targetNamespace filter plus RBAC confines blast radius to chaos-sandbox
- Progressive scenarios: pod-kill, latency, packet loss, partition, degradation