1. About Traffic Distribution

Traffic governance is divided into north-south and east-west. In the typical drawing convention for Network Diagrams, core network components are drawn at the top and clients at the bottom, with different service levels drawn in between — hence the names north-south and east-west traffic.

North-south traffic refers to traffic initiated by clients, while east-west traffic refers to traffic between services. Today, east-west traffic is handled by adding a Sidecar to every service to take over all service-to-service calls, achieving the goal of traffic governance. This post focuses on north-south traffic, that is, gateway traffic.

2. LB Directly Routing to Service Hosts

As shown above, for small-scale businesses we can route client traffic directly to the service through the cloud vendor’s LB, and the service then fetches data from the DB. This is a classic three-tier architecture: an access layer LB, a service layer Server, and a storage layer DB.

The access layer can usually carry a great deal of traffic, and if it needs to scale, it can be split horizontally by business — for example, one LB for a.chenshaowen.com and another for b.chenshaowen.com.

The service layer is stateless, so it can be scaled simply by adding replicas; whether you add VMs or increase the number of Kubernetes Pods, it can be done quickly.

The DB layer is usually the easiest to hit a bottleneck and the hardest to scale. During a business bottleneck, better hardware and a higher-performance database can solve the problem for the short term. But if you want to solve it for the long term, you still have to split. There are two kinds of DB splitting: horizontal and vertical. Horizontal splitting means dividing one table into multiple tables and one database into multiple databases to spread the pressure. For the user volume in China, some user information databases can even be split into hundreds of tables, and at access time the database holding the data can be located according to rules on the user ID. Vertical splitting, by contrast, splits the columns of a table, reducing the number of columns in a single table and the number of tables in a single database.

3. Traffic Distribution Across Multiple Domestic Datacenters

As shown above, as the business keeps growing, we have no choice but to deploy services in multiple datacenters — partly to improve service availability, and partly to improve our bargaining power with cloud vendors.

Before deploying across multiple datacenters, it is advisable to do unitization first. The unitization process is worthwhile: it helps you sort out all of a service’s dependencies and package them into a Unit. Whether it is multiple Units in a single datacenter or multiple Units across datacenters, deploying a new Unit lets you add available replicas, achieve seamless rolling updates, and support features like canary traffic.

With multiple datacenters, user traffic goes through the access layer LB and is distributed to different Units by ratio, then dispatched to specific services through the business gateway in each Unit.

The biggest challenge here is how to guarantee consistency at the DB storage layer. You can buy a multi-active DB instance from a cloud vendor, or of course build your own. Many multi-active DB solutions have high latency requirements. Within a single multi-active instance, the physical distance between instances can reach hundreds of kilometers, which is impossible to satisfy with low latency over ordinary public networks. So the key to multi-active across regions is that you need to build a low-latency dedicated line that interconnects all the datacenters.

Building infrastructure from different vendors’ datacenters has another benefit: each vendor will offer certain discounts, and technical support tends to be much more responsive. Of course, this also requires spending to reach a certain magnitude.

4. Traffic Strategy Across Overseas Multi-Region

The architecture for overseas multi-region is more complex than the multi-datacenter architecture in China. This is because each region has data protection regulations that do not allow local user data to be transferred elsewhere. On the other hand, it also involves the domain strategy: should overseas use one shared domain, or should each overseas region use a different domain?

4.1 A Unified Domain for All Regions

• Routing regions via cookies

As shown above, we can set a region field in the cookies to indicate the user’s region.

Before login, we need to synchronize the user’s region information across regions. At login, DNS resolves to the nearest region, or the LB chooses a region arbitrarily for the login. After login, the region field is set in the Cookies.

Once login is complete, the LB uses the region in the cookies to distribute the user’s access requests to the region where the data lives.

• Routing regions via url

Very similar to the cookie-based routing approach, we can also write the region information directly and visibly into the url, distributing requests to /sg/* to Singapore and requests to /us/* to US West.

4.2 A Different Domain for Each Region

As shown above, using multiple domains is better for user availability and maintainability. In each region, we provide the service with a separate domain — sg.chenshaowen.com pointing to the Singapore service, and us.chenshaowen.com pointing to the US West service.

Before login, GeoDNS geographic domain resolution is used to access the service in the nearest region. If the user belongs to the current region, login completes and redirects to the regional domain. Otherwise, the request is forwarded in the background to another region for login, and then redirects to the service domain of the user’s region.

5. Summary

This post focused on north-south traffic distribution, moving from the classic three-tier architecture to multiple datacenters and finally to overseas regions.

For large services, unitization undoubtedly improves overall scalability, and with the help of traffic gateway distribution it enables features such as failover and canary testing. For overseas business, the first thing to consider is the domain strategy.

In fact, from an SEO perspective, a single domain is much better than multiple domains, since multiple domains dilute a site’s authority. But multiple domains also effectively spread risk: services in each region do not affect one another and can serve traffic independently. If you want to combine the two, you can also use a unified gateway that uses cookies or url to forward to each region’s subdomain — for example, forwarding chenshaowen.com/sg/ to sg.chenshaowen.com.