A Global Image Distribution Network

1. Global Network Planning

Many global, multi-region infrastructures do not put much thought into network planning at the outset. Once the business grows complex enough, they are forced into network adjustments and optimization. Yet any major network change has a huge impact on the business. In the end you are stuck in a dilemma, able to do nothing but invest more people, carry the baggage of history, and walk along the edge of a cliff again and again.

The diagram below shows a network topology I consider close to ideal:

Network planning mainly comes down to the following points:

Subnet segmentation

Under a global business model, the network is split into two parts: overseas and mainland China. I prefer to build two centers — the domestic core node in Beijing, serving mainly domestic business, and the overseas core node in Singapore, serving mainly overseas business.

So 10.128.0.0/16 and above are allocated to overseas, and 10.127.0.0/16 and below to domestic. At the same time, each region’s subnets are spaced 8 apart, leaving room for expansion.

Achieving connectivity

If it is the same VPC, the internal network is reachable. But between different VPCs, different vendors, or different regions, we usually rely on certain methods to achieve connectivity: the public internet or a dedicated line.

The public internet is a fairly universal approach. We can build a VPN overlay on top of it to achieve connectivity. But the quality of public internet connectivity cannot be guaranteed, so there is another option: a dedicated line.

A dedicated line can achieve cross-region connectivity, but a cloud dedicated line is usually limited to a single cloud vendor. That is, a Huawei Cloud dedicated line in Beijing can only connect to Huawei Cloud Singapore, not to AWS Singapore.

Configuring routes

Achieving connectivity is only like plugging in a network cable — when forwarding packets, you still do not know where the next hop of an IP packet is, so you also need to configure routes.

Since there are two network cores, overseas regions need to communicate with the overseas core node, and domestic regions with the domestic core node. Whether other regions communicate with one another depends on whether there is a need. For example, if we need to do P2P distribution of image data over the internal network, then the regions need to communicate with each other too.

2. Building Global Image Distribution Capability

Global image distribution capability is built on the premise that global IDC internal networks are interconnected. We cannot expose our infrastructure to the public internet; all image data is transmitted over internal network traffic.

The diagram below shows a global image distribution system:

Our R&D department is in China, while the deployed services are spread across the globe. Image data flows through the following process:

Images are built in China and pushed to the domestic Harbor.
The domestic Harbor syncs images to the overseas Harbor.
In a given region, an overseas application is deployed and pulls the image.
Because every Docker is configured with Dget’s address as a registry-mirror, the application image is cached in Dget.
Within the same region, when multiple replicas are deployed, they all pull the image directly from Dget.

3. Harbor Deployment and High Availability

3.1 Deploying Harbor

Harbor is mainly deployed in two ways: Helm Chart and Docker Compose. Docker Compose is recommended here, because for a service that changes infrequently and demands high stability, a VM is a better fit than Kubernetes as the infrastructure for Harbor.

3.2 Highly Available Harbor

There are mainly two approaches to Harbor high availability:

Shared storage. High consistency; requires an active-active\active-passive storage backend.
Synchronization between multiple Harbors. Lower consistency; image sync takes time.

I recommend the shared-storage approach: you do not have to wait for Harbor sync to finish — the pushed image is immediately usable. As shown below, under the shared-storage approach the storage components must be deployed in an active-active\active-passive configuration:

There is one small detail about LB configuration:

If you use a Layer 7 LB to offload certificates, the backend host serves port 80, and you need to forward port 80 to 443 at the LB layer, otherwise docker login will fail. If you use a Layer 4 LB, this is not a concern. During debugging I also ran into an issue where both VPN and LB modify IP packets, which can cause odd problems such as being unable to connect or unstable connections. In that case, pay attention to the MTU value.

The components that need to be shared here are:

Shared PGSQL

You can buy the cloud vendor’s service directly, then initialize and create the tables.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
CREATE DATABASE notary_server;
CREATE DATABASE notary_signer;
CREATE DATABASE harbor ENCODING 'UTF8';

CREATE USER harbor;
ALTER USER harbor WITH ENCRYPTED PASSWORD '123456';
GRANT ALL PRIVILEGES ON DATABASE notary_server TO harbor;
GRANT ALL PRIVILEGES ON DATABASE notary_signer TO harbor;
GRANT ALL PRIVILEGES ON DATABASE registry TO harbor;

GRANT ALL PRIVILEGES ON DATABASE harbor TO harbor;
GRANT ALL PRIVILEGES ON DATABASE clair TO harbor;

Just add the external database configuration to the harbor.yaml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
external_database:
  harbor:
    host: 1.1.1.1
    port: 5432
    db_name: harbor
    username: harbor
    password: 123456
    ssl_mode: disable
    max_idle_conns: 10
    max_open_conns: 100
  notary_server:
    host: 1.1.1.1
    port: 5432
    db_name: notary_server
    username: harbor
    password: 123456
    ssl_mode: disable
    max_idle_conns: 10
    max_open_conns: 30
  notary_signer:
    host: 1.1.1.1
    port: 5432
    db_name: notary_signer
    username: harbor
    password: 123456
    ssl_mode: disable
    max_idle_conns: 10
    max_open_conns: 30

Shared Redis

Harbor’s Redis mainly stores user login session information and the sync and scheduled tasks of Job Services. If the availability requirement is not too high, you can use a self-built Redis instance, because even if Redis’s stored data is lost, it has no effect on Harbor’s data integrity.

Shared S3 Object Storage

I use Huawei OBS object storage; the AKSK here needs full permissions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
storage_service:
  s3:
    accesskey: xxx
    secretkey: xxx
    region: ap-southeast-3
    regionendpoint: https://obs.ap-southeast-3.myhuaweicloud.com
    bucket: xxx
    encrypt: false
    secure: true
    v4auth: true
    chunksize: 5242880
    multipartcopychunksize: 33554432
    multipartcopymaxconcurrency: 100
    multipartcopythresholdsize: 33554432
    rootdirectory: /registry/

If you are worried about the single point of failure of S3, you can buy two Buckets and sync image data between them. That way, when one Bucket has problems, you can quickly switch to the other and restore service.

4. Using Dragonfly to Save Bandwidth

Why do we need Dragonfly to distribute images? A big reason is to save bandwidth, and also to avoid overloading Harbor.

Without Dragonfly image distribution, every image pull requests data from Harbor. As shown below:

With Dragonfly, a region only needs to request Harbor once; all other requests can be served by traffic within the region. This greatly speeds up the image pull process, saves cross-region bandwidth, and relieves the load on Harbor.

5. Summary

Recently I have been re-planning and deploying an image management system for our business; this post is a summary of the related thinking and practice.

This article starts with network planning and moves on to global image distribution. Network planning mainly involves three parts: subnet planning, achieving connectivity, and configuring routes. Image distribution mainly uses the Harbor + Dragonfly approach. At the same time, deploying a highly available Harbor with shared storage is recommended.

In fact, after deploying Harbor I also tested the image pull speed in each region. In addition, you need to set up monitoring for the dependencies that affect the Harbor service; only through continuous improvement can you build a good image registry and distribution system.