Code Red - The Data Center Story

"On the occasion of VNG's Data Center collapse with a group of customers completely paralyzed, let us tell you the story of the Code Orange incident that happened to a Cloudflare's datacenter as follows. Hopefully VNG and other vendors start doing the same.

Cloudflare's system scale is much larger than VNG. On November 2, a rare incident occurred: The electrical system caught fire, completely lost power, and the system was paralyzed.

9.00-> 14.00: complete power outage, system paralyzed.

14.00: There is intermittent electricity. Discovered that the entire system was broken and needed to be rebuilt. Each self-healing system takes about 3 hours, there are about 20 systems, total 60 hours.

With systems with standard 99999! then this is impossible.

14.00 -> 18.00: Run all compute in parallel at full speed to build. Not working.

23:00 night: Started running some servers again, but the service is not up.

At 4:25 a.m. on November 4, the service began to be restored.

Total service restoration time: 43 hours.

Code Orange Mission:

What happened happened, the important thing is that after this incident, Cloudflare set an emergency task with code: Code Orange. The mission sets the outcome: that shit should never happen again. Code Orange begins to be designed.

5 months later, the same thing happened: another collapse of the entire data center. At 14:58, the entire data center shut down. Code Orange immediately activates. The entire system immediately enters Self-healing mode.

Miraculously, after just 7 minutes, Code Orange removed the bad data center from the system and miraculously rebuilt the entire service with absolutely no interruption. What used to be a 48 hour loss of service has now turned into a completely normal 7 minutes with only a slight flicker. Code Orange runs completely automatically and does not require anyone to touch it. Total healing time: 7 minutes.

Hopefully VNG will build a similar plan for Code Orange (and even better, Code Red for worse scenarios)."

Author

AiUTOMATING PEOPLE, ABN ASIA was founded by people with deep roots in academia, with work experience in the US, Holland, Hungary, Japan, South Korea, Singapore, and Vietnam. ABN Asia is where academia and technology meet opportunity. With our cutting-edge solutions and competent software development services, we're helping businesses level up and take on the global scene. Our commitment: Faster. Better. More reliable. In most cases: Cheaper as well.

Feel free to reach out to us whenever you require IT services, digital consulting, off-the-shelf software solutions, or if you'd like to send us requests for proposals (RFPs). You can contact us at [email protected]. We're ready to assist you with all your technology needs.

© ABN ASIA