1. Two Years Ago I Took a Different Path

Looking back now, 2021 should have been a golden year for internet workers in Wuhan to change jobs. The pandemic had passed, and we were full of hope for the future; monetary policy was loose, and companies were confident about market prospects.

Against that backdrop, a batch of colleagues working on Kubernetes open-source products jumped ship one after another, heading to cloud vendors to keep working on cloud infrastructure. With solid technical skills and the backing of open-source products, they all landed good offers.

But I did not choose a cloud vendor. I chose a company that leaned toward business. From a technical standpoint, a cloud vendor’s tech stack is more advanced, and technology is their core competitiveness; a business-oriented company’s tech stack is more conservative, and their core lies in the growth of the business. Different goals mean a different starting point when making decisions, and a different strategy for pushing work forward.

The reason for this decision was that I wanted to understand the businesses I supported day to day — how they worked, and why supporting them always surfaced such odd questions. So I simply went straight to a business-oriented company to see how they think and how they make decisions. There was another reason too: from the perspective of social development, technology matters, but business-oriented companies that can fuse technology into production and daily life can bring more happiness to humanity. Inventing electricity was great, but for ordinary people, being able to use electric lights, electric fans, watch television, and use computers is what matters more.

2. What Kind of SRE Does a Business-Oriented Company Need

2.1 Be a Good Firefighter

People who can shorten MTTF (mean time to failure) matter, and business stability takes priority over everything else. An SRE at a business-oriented company needs to be able to locate problems and restore service quickly, and doing that is not easy.

Problems at the infrastructure layer — compute, storage, network — call for depth of technical digging and breadth of technical coverage. Engineers generally stop at a certain point, and when they hit source-code-level bugs they may not be able to hold the line. Traffic is forwarded through a dozen hops at each layer, and every place can have a problem. One node runs a dozen Pods — who knows which Pod will suddenly go crazy and hammer IO.

Problems at the platform layer require deep familiarity with SRE-related facilities. You know how to write PromQL, but you cannot find the addresses of Grafana or VictoriaMetrics; you see a monitoring dashboard, but some of the metrics in it are drawn wrong; you suspect uneven load is the cause, but you cannot find the monitoring and logs to prove it; you want to look at Kubelet logs, but you cannot log in to the operating system. I believe everyone can master generic SRE skills, but each company’s usage, practice path, and level of mastery differ, and it takes time to become familiar enough to locate problems quickly. Collaboration with other colleagues takes adjustment, and building trust takes time too.

Problems at the application layer require being able to help the business land best practices, and ideally to read the business code. When there is an incident, the one who takes the blame is most likely the SRE. If you are not convinced and cannot find the cause, you have to go read the business code. Maybe after reading it a few times and talking with the business a few more times, you find the problem; maybe you flip through the code until it is worn out and are still completely baffled, and can only resign yourself to fate and pray that this kind of problem does not recur.

The firefighter is the role with the strongest presence on the team — highest in both contribution and influence. Being a good firefighter is a solid entry point for establishing yourself on an SRE team.

2.2 Build Products with an Entrepreneurial Mindset

To establish yourself as an SRE at a business-oriented company, you need something to stand on. Besides technology itself, you also need SRE products. A product is an amplifier of technology: an individual’s technology can only serve a limited audience, while a product can serve more of them without significantly increasing marginal cost — which is exactly the kind of thing the boss likes to see.

Fighting fires all day is not a long-term plan; only with a product you lead does the SRE have room to grow. Upward, there is a clear OKR and KPI direction that demonstrates value; downward, there are more work items and room to exercise your abilities.

Why do I say you should build products with an entrepreneurial mindset? An SRE at a business-oriented company has no well-established workflow. An SRE should be a generalist: we can define SLIs and guarantee SLAs; we can be developers and we can be operators; we can be product managers and we can be operations people. In short, we need to care about the entire lifecycle of a product.

But the hard part of entrepreneurship is not technology or product — it is finding customers. At a business-oriented company, the customers are the business teams. The needs of business colleagues are often vague, even indescribable. This requires the SRE to proactively dig out requirements, proactively communicate with business colleagues, and proactively push the product toward landing.

The whole process may last a long time — months, a year or two, or even longer. Throughout it you need to continuously report upward to show results and win support, and push downward to build good examples and ensure the product lands. This demands strong execution, sales ability, and stress tolerance from the SRE.

2.3 Blooming Where the Weeds Grow Thick

We talked above about the mindset for building products; in this part let us talk about the products themselves.

The basic functional demands in the operations domain are release, build, monitoring, logging, alerting, CMDB, ticketing, and so on. These core topics have been studied to death by our predecessors, and it is nearly impossible to land a product from 0 to 1 cleanly according to your own ideas.

Best practices are all over the internet, but to land them internally, execution matters far more than imagination. You can take slides around and present them, but when it comes back to specific matters, you still have to advance a little each day and do it bit by bit.

The weeds here are not meant to disparage the existing systems — they just represent the many distractions, historical burdens, and new requirements. These are extremely common problems in actual work. Without a leader with the boldness to guide you and the unspoken coordination of business colleagues, it is very hard to produce results.

Refactoring an existing product is fairly difficult, so we need to focus on increments, which also fits the business’s need for rapid growth. Design for the future; do not be troubled by the weeds in front of you, and let the old systems slowly degrade until they are decommissioned.

3. Opportunities and Challenges

3.1 Growing Along with the Business

Compared with an SRE at a cloud vendor, an SRE at a business-oriented company has a big advantage: a larger expansion coefficient.

For the same amount of expansion, a business-oriented company adds more technical staff. A business-oriented company does not consider scalability the way a cloud vendor does. Considering scalability increases design difficulty and raises development and operations costs. The goal of a business-oriented company is the growth of the business, not the growth of technology. This differs from a cloud vendor’s way of thinking.

When the business grows, the reserve of technical staff grows, which creates opportunities for promotion. When a business-oriented company makes money, there are opportunities for a raise. Only then can we live better lives.

3.2 Testing Your Own Methodology

At a platform company, employees easily mistake platform capabilities for their own abilities. If the platform product is great, they feel they are on the same level — but that is not the case.

Knowledge and methods need to be internalized. After we hear, see, and remember, what matters more is to practice them ourselves, to summarize, to distill, to form our own methodology — a self-consistent and complete logic.

For a technical person, once you have formed your own thinking, you should make it concrete and bind it to yourself, just like https://www.chenshaowen.com/ops — aim for the optimal solution rather than writing defensive code. This is in fact a win-win process with the company: we build tools that make everyone’s work more efficient, and the open tools also make us more competitive in the market.

Standing on your own and accumulating internally — opportunities like this are in fact hard to come by at a big company.

3.3 Technical Regression

Compared with a cloud vendor, a business-oriented company focuses on the tech stack the business uses and does not pay much attention to the evolution and development of technology.

Because the customers a cloud vendor serves are more diverse, it sees a more complete technology cycle and is more forward-looking — which is also one of a cloud vendor’s selling points. They can not only provide cloud services, but also provide technical backstopping, and can even serve as the company’s technical advisor, pointing out the technology roadmap for the next three to five years.

It is precisely this selling point that forces a cloud vendor to study every detail and every possible defect in depth. But a business company does not necessarily need to solve these hard problems — we can waste a bit of resources and write somewhat inelegant code to work around them.

Another aspect is the technical atmosphere. What an SRE at a business-oriented company discusses day to day is closely tied to the business. The technical roadmap will not run ahead of company strategy; it can only run slightly ahead of business planning.

The problem of technical regression for an SRE at a business-oriented company is hard to avoid, but it is also a kind of trade: we gain a deeper understanding of the business. Whether this trade is worth it depends on whether the business domain has prospects and whether there are barriers to entry.

3.4 Frequent Personnel Changes

When anxious, small and medium-sized companies may keep reorganizing their structure and shuffling personnel.

When people change, the matters change, and the goals change: what was done this year will not be done next year. Work without continuity is dispiriting for anyone who does it, offers no sense of accomplishment, and easily breeds an attitude of slackness.

If you are coasting wherever you are, frequent changes are actually fine; but if you really want to get something done and build up some accumulation, then taking it seriously means you lose.

What notably distinguishes operations from R&D is that operations must be steady while R&D must be fast. With clear goals and long, sustained investment, an operations system can be built very well, because the core problems are very clearly defined; R&D, much of the time, is betting on whether the market will explode — it only adds icing to the cake, never delivers coal in the snow.

4. Summary

This article is mainly about some of my experiences over the past two years as an SRE at a business-oriented company. It is written mainly from the SRE’s perspective, discussing what an SRE at a business-oriented company needs to do, and what opportunities and challenges they will encounter.