1. Only a Wheel Turning at High Speed Stays Balanced

1.1 The Consumer-Side Boom Is Over

As of June 2023, China had 1.079 billion internet users, up 11.09 million from December 2022, with an internet penetration rate of 76.4%. The consumer-side demographic dividend has passed, time spent online is also growing slowly, and there is little room left for growth in applications built on existing scenarios.

After the easy mode of the consumer market, many companies cannot adapt to the hard mode of the business market. On the consumer side there is a large enough crowd to let us polish our products, and a wide enough variety of monetization methods; but business users give us no chance to experiment at all — they demand a perfectly fitting product right away, and to control marginal cost, software products invest very little in supporting non-standard items. Taking the money earned on the consumer side and pouring it into the business side to find new growth points is exactly what many big tech companies are doing today.

1.2 The Building Is Finished; So Many Workers Are No Longer Needed

This is a point a colleague and I talked about during a business trip in 2023: our platform is already built, like a finished high-rise — what it needs is a small property-management crew, not a large construction workforce.

When our efficiency platforms and business services were developed from 0 to 1, we hired experts from outside at high cost and mobilized internal key staff to build them eagerly; during the 1-to-100 iteration we kept pouring in large amounts of machine resources, operations manpower, and high response priority; but what about 100 to 110? Marginal returns keep shrinking, and even investing huge resources makes linear returns hard to obtain.

The platform is done, and small patches are enough to meet demand. Where two people once maintained one product, now one person maintains two. Even mutual backup staffing is unnecessary — it is like wanting to hold a hilltop with a single machine gun, with no expectation that any enemy will come.

1.3 Growth Slows, and Stability Problems Come to the Fore

When the business is growing fast, the product iterates constantly and someone is watching production, so problems can be found and fixed in time. But when growth slows, the product loses a steady release cadence and the people involved lose their sense of responsibility, so all kinds of small production issues get ignored — and the concentrated eruption of these small issues is what causes the truly major incidents.

The reverse is also true: a major incident hides many small problems. Behind a single unavailability incident there may lie many defects, for example, no disaster recovery, no failover, no rate limiting, no monitoring, no alerting, and so on. If every one of these were done well, with no weak spots, the incident very likely would not have happened.

2. What Comes Next Is the Real Challenge

In 2023 I lived through many business incidents, large and small, including ones that trended on Weibo.

2.1 Is Our Engineering Solid Enough

One Goroutine leak incident after another, each different. Packet loss — goroutines blow up; cannot connect to the database — goroutines blow up; not enough CPU — goroutines blow up.

Foundation libraries are old and of many different versions, the Golang version is low, and there is not even a unified application framework.

These exposed problems are already hard to solve in an era of no growth. But production problems keep coming and have to be solved, so the time has come to test how solid our engineering really is.

2.2 Is the Architecture Sound

Service calls are tangled and complex, with no governance. The complexity a person can handle is finite; during the growth years we let teams choose their own stacks in a laissez-faire way, with no unified management, and no one can map the service call relationships clearly.

Cross-cluster calls, cross-datacenter calls, internal calls going over the public internet — a mess of all kinds — and a few services calling one another multiplies the QPS several times over.

Local-first service calls, local-first user access, unitized cluster design — problems that were once ignored now directly affect application stability.

2.3 Is Stability Reliable

The increment is gone, so 90 points is out of reach; if you want to coast at a passing 60, you still have to defend the stability of the existing business.

That is how things usually go: aim high and you land in the middle; aim for the middle and you land low. Simply holding the status quo is no easy task. Many exceptions used to be handled by hand, and now that the people have been laid off, that handling capability is gone too.

But the SLA still has to be met, which means we must find new methods to guarantee stability. Chaos engineering experiments are an excellent method: by deliberately creating controlled failures, they improve the stability of the system.

3. Internet Applications Have Chaotic Characteristics

3.1 The Birth of the Discipline of Chaos

Before discussing chaos, we can first think about the connections among chaos, chaos engineering, and our production services.

The story we often hear is that a butterfly in the Amazon flapping its wings a few times can set off a tornado in the United States. Behind this story lies an important discipline, and that is chaos.

As early as the 1960s, Lorenz discovered the phenomenon of chaos. He was a mathematician, but worked in meteorology, and while studying weather with mathematical models he found that a tiny change in an initial value leads to a huge difference in the result. Extreme sensitivity to initial values is an important characteristic for judging a chaotic state. After more than a decade of further development and research, this sensitivity to initial values was named the butterfly effect and became widely known.

3.2 The Birth of Chaos Engineering

The chaos engineering we are talking about, however, actually refers to engineering practice that came out of Netflix in 2008.

In the course of Netflix’s move to the cloud, they discovered a problem: after migrating, service stability declined. The reason is that the network, machines, storage, and so on are all uncontrollable.

So they developed a tool called Chaos Monkey that proactively breaks cloud services to expose the system’s weaknesses and thereby improve its stability. Today it is an open-source project with tens of thousands of stars, https://github.com/Netflix/chaosmonkey .

Like a monkey let loose in the data center, it randomly breaks the system, and you discover the fragile points of your system as well as learn handling experience from failures.

3.3 The Relationship Between Microservices and Chaos

Today’s internet applications are mostly microservices, whose defining trait is that the dependencies between services are very complex. So do microservices have chaotic characteristics?

Here are the similarities and differences I have summarized.

Similarities:

1. Complex, nonlinear connections
2. Sensitive dependence: a change in one microservice affects the other microservices associated with it, similar to chaos’s sensitive dependence on initial conditions
3. Possibly huge accumulated error, similar to the butterfly effect

Differences:

1. Uncertainty: microservices < a typical chaotic system
2. Microservices emphasize independence, whereas chaotic systems emphasize mutual dependence

A computer system is a deterministic system: the program code is written in advance and the running result is expected. But in reality, is a computer system really deterministic? No. Disk damage, memory bit flips, network interruptions — the physical world in which a program runs is uncertain. In addition, the resources of microservices are not exclusive; under high-density deployment, CPU, memory, network, and other resources are all shared, and services influence one another. Here I will call all of this the environmental factors in which a program runs.

From the perspective of the computer system + environment, our production systems and microservice programs do have chaotic characteristics.

4. The Necessity of Running Chaos Engineering Experiments

4.1 It Provides New OKR Objectives

If it is not in the OKRs, it is not important. If we want to hold the existing business steady, we must not only know the existing system problems well but also make those problems visible and recognized through OKRs.

When leadership has items to list and the team has things to do, only then can we ensure that our work has a path to land.

4.2 Escorting the 2.0 Transformation

The existing business cannot aim only for a passing 60; we still need to keep trying new technologies and new architectures.

In that process of experimentation we need a protective mechanism to ensure our business is not affected. Chaos engineering experiments are an excellent way to break the deadlock, and an excellent way to validate as well.

For the existing architecture, we can bring chaos engineering experiments to bear on it, guiding it to evolve rather than endlessly maintaining it.

4.3 The Environment Applications Run In Is Growing More Complex

The infrastructure environment and software architecture we live in grow ever more complex, and failures have become inevitable.

In the classic architecture, the IaaS layer may see service outages and network jitter; the PaaS layer may see database outages, uneven load, and CPU contention; the SaaS layer may hit OOM, CPU throttling, inability to connect to the database, and other failures.

The complexity of an application and its runtime environment now far exceeds what an individual can fully grasp. Rather than waiting for failures, we should go on the offensive and create them ourselves.

4.4 Reproducing Failures Is Getting More and More Expensive

“I suspect the CPU is not enough”, “I suspect there is a network problem”, “I suspect we cannot connect to the database”, “I suspect….”

When we suspect some uncertain factor, we should not stop at the guessing stage; we need a chaos engineering experiment.

Once we have the methods and tools of chaos experiments as support, we can verify our guesses through experiments instead of troubleshooting blindly.

5. Where the Value of Chaos Engineering Lies

5.1 Accumulating Failure-Handling Experience

Having a plan in advance is far better than improving after the fact.

The common practice is that after an incident occurs, we hold an incident review, distill some failure-handling experience, and then codify it into a failure-handling manual. But you may find that the next incident is a new one, and the previous experience cannot be fully applied.

This is because the sample of incident-handling experience is too small and the system is too complex, whereas frequent chaos experiments can enrich the sample of failure-handling experience and let us respond better to unknown failures.

5.2 Verifying Existing Defects and Discovering Unknown Defects

Chaos engineering can expose unknown defects in production systems and improve their stability.

One point must be made clear here: if it is already certain that the chaos experiment will cause problems, that means the existing system has no fault tolerance or response mechanism at all, and so such an experiment is meaningless.

If we divide by the boundary of what we know, defects can be split into two categories:

1. Known defects, including those already tolerated and those known but unrepaired, such as single points of failure, network jitter, and the like
2. Unknown defects, including the unknown and the misjudged, such as latent bugs and defects we believe we do not have but actually do

For defects inside the boundary of what we know, chaos experiments can be used to reproduce defects, assist in fixing defects, and verify that defects have been fixed.

For defects outside that boundary, chaos experiments are exploratory. Through chaos experiments we can discover defects that were overlooked or left out.

5.3 Improving Application Resilience

While studying various chaos engineering practices, I also came across an interesting concept: application resilience.

When I first encountered this concept it always felt familiar, yet hard to articulate. So I thought of a vivid analogy: like the roly-poly toy in the image above, an application can sway left and right but still keep its balance. This requires the application to heal itself when facing all kinds of unstable, uncertain, and sudden situations — after a brief wobble, to return quickly to a balanced state.

To position an application as a roly-poly, the following designs are needed:

• Redundant design

For critical middleware, two-region three-datacenter, tolerating at least one of them being disconnected

• Overload protection

When requests surge, be able to use rate limiting, service rejection, and other measures to protect the whole from damage

• Service degradation capability

When service capacity is insufficient, be able to make trade-offs and ensure key services are not affected

• Decentralized design

Centralized nodes are not allowed

These concrete requirements have a path to implementation for our business applications. We can use this requirement to assess and rework our applications.