1. Choosing a Communication Protocol
Ops systems are better suited to HTTP than to gRPC.
More ops and R&D engineers are familiar with HTTP than with other protocols. Once you know the HTTP protocol, learning a RESTful-style HTTP API is quick. Being familiar to more people and easier to learn means better communication and lower handover cost, because they share more common domain background.
There are a great many tools that support HTTP debugging. Whether it is a browser, various plug-ins, or command-line tools, you can debug an interface very conveniently.
System integration is convenient. Ops people are good at scripting languages such as Shell and Python; calling an HTTP interface is easy, with no need to generate a gRPC client and program against it. A low cost of system integration means the service you provide will be used more.
For an ops system, a communication protocol that is clear, simple, and well compatible matters more, and performance need not be emphasized β a marked difference from business systems.
2. Centralized or Decentralized Data Management
Ops systems are better suited to centralized data management than to decentralized data management.
You can think of the ops system as the control plane of the whole company’s systems, and the business system as the data plane of the whole company’s systems. The control plane carries control instructions: how applications are deployed, how traffic is routed, how to handle a service exception, and so on. The data plane carries the data of business users: browsing pages, downloading images, uploading videos, and so on.
Ops systems have fairly high requirements for permissions, security, auditing, and so on, and centralized data management satisfies this well. Business systems, however, store user data, which for compliance reasons cannot be transmitted across countries or regions, so it can only be stored in a decentralized, local way.
It is worth stressing that an ordinary ops system does not have much concurrency; a company with 10K machines is already quite substantial. If you only consider human use, an ops system that can support 1K PV is enough to meet the needs of the vast majority of companies. Ops systems have much lower scalability requirements than business systems.
3. The CAP Trade-off
An ops system chooses CP, sacrificing some availability; a business system chooses AP, sacrificing some consistency.
Ops systems have higher consistency requirements: even if the interface responds a little more slowly, they must synchronously complete all kinds of checks before returning from the interface and confirming the call succeeded. Otherwise they should roll back immediately and prompt the user.
Business systems, by contrast, care more about user experience; in most scenarios they can return the result directly and throw the task into a queue for asynchronous processing.
4. Simplicity Beats Everything
When developing an ops system, you should design on the basis of the fewest constraints and rules, and only then consider other details.
An ops system is far more complex than a business system: an ops system is ToB, while a business system is ToC. A ToC product ultimately targets the whole population, whereas a ToB product ultimately targets a vertical domain and requires reading documentation and repeated experimentation to become proficient. So ToB is usually accompanied by a training market that guides ToB users, answers their questions, and provides solutions.
But simplicity in an ops system is not easy; it has to be raised to the level of a design philosophy, and that philosophy must always be able to guide the design of the system. This places high demands on the team; otherwise the whole product system will look chaotic, and users will have no idea where to start. Imagine that a change is needed and one moment it is imperative and the next declarative β how confused users would be.
So from the very beginning of designing an ops system, simplicity must come first, in order to offset the complexity introduced by satisfying business requirements.
For an ops system, even an ESB architecture is sufficient; there is no need to emphasize microservices, let alone a service mesh. An increase in complexity is the root of all kinds of problems.
