1. Quota Limits

Every resource in the cloud has a cap on how much of it you can use.

This is how cloud vendors prevent resource abuse and reduce the mutual impact between tenants.

For example: the number of enterprise projects under one account, the number of elastic compute instances, the number of elastic public IPs, the bandwidth of an elastic public IP, and so on.

Resource requests failing because of insufficient quota are very common; but during a peak season for elastic workloads, this problem becomes serious.

If you have built your own control plane or an internal cloud management platform, you need to pay special attention to these quota limits and monitor them well.

2. Performance Problems Caused by Sharing

Some cloud resources come in dedicated and shared flavors, and shared is much cheaper than dedicated.

Many times we choose shared resources out of cost considerations, but end up digging a pit for ourselves.

After Huawei Cloud’s shared load balancer enables performance assurance mode, it provides a guaranteed capacity of 50,000 concurrent connections, 5,000 new connections per second, and 5,000 queries per second. If these limits are exceeded, the business side will start throwing errors.

Shared resources of any kind can run into similar problems: the SLA cannot be guaranteed, or the performance cannot be guaranteed.

3. Forced Resource Upgrades

To fix version vulnerabilities and keep up with new features, cloud vendors forcibly upgrade some resources.

Cloud vendors’ forced upgrades are usually hot updates, and they notify users in advance — but users do not necessarily pay attention.

A forced upgrade is in fact a responsible act, and it requires the user to keep investing effort to follow along; yet users often ignore the matter, considering it thankless.

Overseas vendors, such as AWS, forcibly upgrade databases like RDS, and these database upgrades cause a brief disconnection that makes the database unreachable for a short time. The EKS upgrade, by contrast, is done by replacing the underlying EC2 instances, and it is very smooth.

Paying attention to cloud vendor notifications, intervening actively, and taking the initiative to upgrade is far better than being passively upgraded.

4. Parameter Differences Between Cloud Vendors

Unlike self-built infrastructure, which is uniform, cloud vendors’ infrastructure parameters and configurations vary a great deal.

For example, after you reboot a Huawei Cloud host, NetworkManager overwrites the DNS configuration in /etc/resolv.conf. The real cause is that this is not the right place to configure DNS, but we never hit this problem on other cloud vendors.

Another example: AWS supports jumbo frames, so the MTU can be set to 9001, but Huawei Cloud does not.

The advice is to customize the configuration of your infrastructure rather than relying on cloud vendors’ default configuration, including network, storage, security groups, and base images. Pinning these configurations down in advance is far better than aligning them after the fact.

5. Cloud Vendor Hardware Failures

If you are a large customer, you will most likely often receive notices like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

Dear xxx user:
Hello! The following RabbitMQ instances in region xxx under your account need to be migrated due to a failure of the underlying physical machine.
Impact: 1/3 of long connections disconnect briefly, and some messages not consumed in time are lost.
Please be informed. Instance information is as follows:
UID: xxx
Region: xxx
Product: RABBITMQ_GROUP
Instances with an owning project:
Project name: default project
   Instance ID: xxxx
   Instance name:
   Subtype: xxx

Cloud vendors have hardware failures quite often, AWS especially, since its machines are really too old. Although cloud vendors have redundancy and live migration, when a failure happens they cannot guarantee 100% immunity, and the business will notice.

When choosing resource specifications, use multiple AZs if you can, and multiple Regions if you can. Core business should trade money and redundancy for stability.

6. Cloud Vendor Software Failures

Cloud vendors’ code is also written by humans, and wherever there are people, there are BUGs.

This is a recent incident report:

1
2
3

Root cause of the failure: a code defect.

Trigger scenario: when creating a scaling group activity, the system needs to deliver cloud host information, create instances, query instance information, and sync ES. Because this process took too long, the Redis data was written later than the time when the 5 cloud hosts in the scaling group finished being created (multiple interfaces took longer than expected, and the cumulative effect made the overall time exceed expectations). After the console receives the power-on completion message from the cloud platform, when handling the scaling group logic it needs to fetch the corresponding information from Redis...

The cloud vendor’s attitude was very good, but that does not solve the problem — you can only try to minimize similar incidents. The less of a top-tier vendor it is, the more software problems there may be.

Build SLO metrics for cloud vendor functions. For core functions — those that directly affect business stability — build a monitoring and alerting system, and report to the cloud vendor immediately for intervention the moment an issue is found. Otherwise, by the time the business has a problem, the situation becomes very urgent.

7. Arrears Preventing You from Creating Cloud Resources

You would not expect it, but there is also the pit of being unable to create cloud resources because of arrears. Arrears do not affect existing resources, but they do affect creating new ones.

I have personally been through two incidents of being unable to create cloud resources due to arrears, one with a domestic account and one with an overseas account; overseas payment cycles are relatively long.

The cloud vendor notified us the night before: The bills for March, April, and May are unpaid; at present account xxx is in arrears; our commercial team is already applying for a credit limit expansion, which will take 2 days;

The next day, elastic scaling failed because of the arrears, and the whole morning everyone was chasing the company to wire the money, chasing the cloud vendor to expand the credit limit, and shifting business traffic. It was quite a waste of operations manpower.

Public-to-public reconciliation cycles are long, so arrears situations arise easily, and raising the credit limit without any cap is certainly not feasible. Every time you are in arrears and have the cloud vendor handle it urgently, you degrade the quality of service the cloud vendor gives you.

The advice is to negotiate with your company’s finance department and the cloud vendor to pay on schedule, and to have a credit limit that covers one cycle. If the vendor sends a payment reminder, do not delay — wire the money promptly.

8. Human Operational Errors During Cloud Vendor Changes

The cache service lost connectivity, and all the related services became abnormal. The cloud vendor explained that while moving other equipment, the four core production switches were operated by mistake at the same time.

“Shutting down four core switches at the same time,” whether it was done by an outsourced worker or not, is a massive process failure on the cloud vendor’s part.

Not long after, another cloud vendor also caused a large-scale service outage due to a human operational error.

The reason was that while modifying a bandwidth configuration — it was supposed to go from 40Gbps to 50Gbps — the operator made a mistake and changed it to 5Gbps, causing a large-scale service outage.

These explanations from the cloud vendors are absurd, though it is also possible that they are covering up some even more elementary mistakes.