Two months ago, I gave a talk to a business team about chaos engineering in practice. This post mainly organizes the content of that talk.

Click to view the presentation

1. The Origin of Chaos

1.1 The Birth of the Discipline of Chaos

Before discussing chaos, let us first think about the relationship between chaos, chaos engineering, and our production services.

The story we often hear is that a butterfly flapping its wings in the Amazon can cause a tornado in the United States. Behind this story lies an important discipline: chaos.

As early as the 1960s, Lorenz discovered the phenomenon of chaos. He was a mathematician who worked in meteorology, and while studying weather with mathematical models he found that a tiny change in the initial values leads to an enormous difference in the result. Extreme sensitivity to initial values is an important characteristic for judging a system to be chaotic. After more than a decade of further development and research, this sensitivity to initial values was named the butterfly effect and became widely known.

1.2 The Birth of Chaos Engineering

The chaos engineering we are talking about actually refers to the engineering practice that came out of Netflix in 2008.

In the process of moving to the cloud, Netflix discovered a problem: after moving to the cloud, service stability declined. The reason is that the network, machines, storage, and so on are all uncontrollable.

So they developed a tool called Chaos Monkey that actively breaks cloud services in order to find the system’s weak points and thereby improve stability. Today it is an open-source project with over ten thousand stars: https://github.com/Netflix/chaosmonkey .

Like a monkey let loose in a data center, it randomly breaks the system; you discover the system’s fragile points, and you also learn how to handle failures from them.

1.3 The Relationship Between Microservices and Chaos

Today internet applications are mostly microservice-based, and the defining trait of microservices is that the dependencies between services are extremely complex. So do microservices have the characteristics of chaos?

Here are the similarities and differences I have summarized.

Similarities:

1. Complex, non-linear connections
2. Sensitive dependence: a change in one microservice affects the other microservices it is associated with, similar to chaos’s sensitive dependence on initial conditions
3. Potentially enormous cumulative error, similar to the butterfly effect

Differences:

1. Uncertainty: microservices < typical chaotic systems
2. Microservices emphasize independence, while chaotic systems emphasize mutual dependence

A computer system is a deterministic system: the program code is written in advance, and the result of running it is expected. But in reality, is a computer system really deterministic? No. Disk failures, memory bit flips, network outages — the physical world in which a program runs is not deterministic. In addition, the resources of a microservice are not exclusive: under high-density deployment, CPU, memory, network, and other resources are all shared, and services affect each other. Here I will refer to all of this collectively as the environmental factors of program execution.

Viewed as computer system + environment, our production systems and microservice programs do have the characteristics of chaos.

2. Chaos Engineering — Providing the Technical Means

2.1 The Necessity of Running Chaos Engineering Experiments

The infrastructure environment and software architecture we live in are becoming ever more complex, and failures are already unavoidable. Looking at the classic architecture: the IaaS layer may have service outages and network jitter; the PaaS layer may have database outages, uneven load, and CPU preemption; the SaaS layer may encounter OOM, CPU throttling, inability to connect to the database, and other failures.

In fact, the complexity is already far beyond what any individual can fully grasp. Rather than waiting for failures, we should take the initiative and create them ourselves.

“I suspect it’s not enough CPU,” “I suspect there’s a network problem,” “I suspect it can’t connect to the database,” “I suspect…”

When we suspect some uncertain factor, we should not stop at the guessing stage; we need a chaos engineering experiment.

2.2 Chaos Features Integrated into the Platform

It is mainly divided into two parts:

• chaos-agent

For fault injection into general-purpose infrastructure, the scenarios are very generic, so I chose the fault probes extracted from chaos-mesh. This way, fault injection at the IaaS and K8s levels can be used directly, while for things specific to our own business — such as fault injection into the internal DB PaaS or into IDC dedicated lines — we need to develop probes ourselves.

• chaos-controller-mgr

A self-developed K8s-based Operator that defines two objects, Network and Stress, and provides three kinds of fault injection capability. There were two considerations for not using chaos-mesh’s control plane directly. One is risk control: chaos-mesh’s controller has to handle too many injection types, and we want to avoid accidents caused by improper use. The other is to make it easy to combine with our own probes — a single control plane can unify both infrastructure and business fault injection.

2.3 Feature Overview

It is currently deployed in the test cluster and provides three kinds of fault injection capability.

• CPU stress
• Network packet loss
• Network disconnection

2.4 Injection Principles

The control plane’s workflow:

1. Submit a fault injection request
2. The controller processes the request, finds the agent on the node where the Pod runs, and makes an RPC call
3. The agent finds the container’s namespace and writes the fault on the node
4. When the timer expires or injection is paused, clean up the fault

Injection principles:

CPU - The Stress command enters the process namespace to inject
Network - Iptables and IpSet commands hijack the network namespace’s traffic, controlled via TC

3. Where the Value of Chaos Engineering Lies

3.1 The Benefits It Can Bring

Chaos engineering can expose unknown defects in production systems and improve system stability.

One point that needs to be made clear: if it is already certain that a chaos experiment will cause problems, that means the existing system has no fault tolerance or response mechanism at all, and such an experiment is meaningless.

If we divide by the boundary of what we know, defects can be sorted into two categories:

1. Known defects, including those already tolerated and those known but not yet fixed, such as single points of failure and network jitter
2. Unknown defects, including unknown defects and defects involving cognitive bias, such as latent bugs and defects that we assume are fine but that do not match reality

For defects inside the boundary of what we know, chaos experiments can be used to reproduce them, help fix them, and verify that they have been fixed.

For defects outside that boundary, chaos experiments are exploratory. Through them we can discover defects that have been ignored or overlooked.

3.2 Steps for Implementing Chaos Engineering

To make it easier for everyone to put this into practice and operate it concretely, here is a simple set of implementation steps.

1. Define steady-state metrics: throughput, error rate, latency percentiles, and so on
2. Expected hypothesis: assess the behavior according to the relevant people’s understanding of the system
3. Run the chaos experiment: by some means, inject the fault into the system
4. Observe whether the steady-state metrics match expectations and record the data
5. Recover from the experiment, summarize and restore the scene, fix the problems found, and continue experimenting

On this basis, we have a convention for every chaos experiment we run. Ideally the chaos platform should provide this capability, letting everyone manage faults, record phenomena, define response strategies, conduct postmortems, and so on, supporting the entire lifecycle of a chaos engineering experiment.

But no such platform has been built internally yet, so I strongly recommend first recording everything in this flow as documents, and later consolidating it onto the platform.

3.3 The CEMM Maturity Model

For chaos engineering and stability, CAICT has proposed a standard called CEMM, and we are building our chaos engineering platform against Level 3. A few brief points:

The results of chaos experiments should feed back into the application’s health status metrics. This requires us to integrate chaos engineering experiments into the application lifecycle and establish an evaluation mechanism, so that an application must pass the test of chaos engineering experiments before it can be released.

Self-service experiment creation and automatic experiment execution. We have actually already met this requirement, but the current form of experimentation is fairly crude and lacks management of the complete experiment process.

Experiment results can be collected continuously through tooling, but they require manual analysis and interpretation. Before I developed the chaos injection feature, we were conducting various chaos engineering experiments manually, lacking automation and tool support. Having accumulated experience, the next step is — strongly recommended — to strengthen tool support: fault injection tools, metric collection tools, result analysis tools, exception handling tools, fault recovery tools, and so on.

3.4 Application Resilience

While studying various chaos engineering practices, I came across an interesting concept: application resilience.

When I first encountered this concept, it felt familiar yet hard to pin down. So I came up with a vivid analogy: like the roly-poly toy in the image above, the application can sway left and right but still keep its balance. This requires that when facing unstable, uncertain, and sudden situations, the application can heal itself, and after a brief fluctuation quickly return to a balanced state.

To position the application as this kind of roly-poly, the following designs are needed:

• Redundancy design

For important middleware, a two-region three-datacenter setup, allowing at least one of them to be disconnected

• Overload protection

When requests surge, be able to use rate limiting, request rejection, and other measures to protect the whole from damage

• Service degradation capability

When service capacity is insufficient, be able to make trade-offs and ensure that key services are not affected

• Decentralized design

No centralized node is allowed to appear

These concrete requirements have a practical path for our business applications. We can assess and rework our applications according to them.

4. Next Steps — Implementation Through Joint Effort

• Establish application quality standards and build consensus around them

We need the ability to sense anomalies. Only when we can sense them can we respond and take further action. If an application is malfunctioning and nobody pays attention, that is certainly not acceptable.

Besides sensing anomalies, we must also be able to tolerate a certain amount of them. An application should not crash the moment there is packet loss; it should have a certain fault tolerance.

• Regular, exploratory chaos engineering experiments

Some time ago, we ran continuous randomized chaos engineering experiments in the test cluster. We randomly picked applications and randomly chose fault injection types and parameters, and found quite a few problems.

• Platform and business teams jointly build resilience improvement plans

The chaos engineering platform team hopes to keep following up on a number of application resilience improvement cases and promote them as classic examples. These accumulated plans are the best practices for resilience improvement suited to our current infrastructure and business forms.