Feb 22, 2021 Spot/Margin system service failure report

1. System downtime description:
From 3:10 am to 6:10 am UTC on Feb. 22, OKX detected abnormal behavior in terms of Spot and Margin trading, affecting the web, app, and API servers.
After troubleshooting, it was discovered that the abnormal behavior was a result of a bug which caused the internal services, that the trading system relied on, to stop working.
OKX engineers responded urgently and managed to resume all functions by 6:10 am UTC on Feb. 22. The timeline for this incident is detailed below:
3:10 am UTC: Our detection system discovered an abnormal condition in the system and issued an alarm message.
3:10 am UTC: The error code "30030" was returning with a "Matching engine is being upgraded. Please try in about 1 minute" prompt on the API trading interface. Spot trading and margin trading service were suspended.
3:11 am UTC: Urgent repairs were initiated.
3:15 am UTC: OKX engineering team found that the failure resulted from the internal service's sudden unavailability, which affected the spot and margin trading system.
3:31 am UTC: Most of the trading services were recovered. Some users intermittently faced failure to request interfaces since one of the servers was unable to reload.
6:10 am UTC: All trading servers were recovered.

2. What work do we do to ensure the stability of the OKX platform?

OKX provides 24/7 trading services and has been dedicated to making its trading system ultra-stable and smooth. However, given the complexity and unexpected abnormalities of a trading system with high performance, we cannot guarantee that the system will work perfectly at all times. However, we have been working hard to improve system stability and minimize the probability of downtime from all aspects, including:

1). We strengthen engineering quality assurance and optimize the test system. The code for new functions can be launched only after it runs stably for a period of time in demo trading.
2). We upgrade architecture. The high availability of multiple servers in various regions is being realized, with less downtime caused by hardware and software problems.
3). Hot upgrades will be realized in a stateless way, which reduces the impact of the upgrade on user transactions.

3. How to get updates from OKX?

(1) Once we detect failures, we will immediately publish failure notifications on the Status page.
(2)If there is any system upgrade scheduled, we will publish a notification on the Status page and notify users via market and community channels (API user community + regular user community). Meanwhile, API users can be notified of the updates by subscribing to System/Status channel.