RocketMQ Reaches Top-Level Status at the Apache Software Foundation – InApps 2022

RocketMQ, the distributed messaging and streaming platform that enabled Alibaba to handle 1.2 trillion concurrent online messages during its Black Friday-like Global Shopping Festival last November, has become a top-level project with the Apache Software Foundation.

The distributed messaging and streaming platform was created for Alibaba‘s e-commerce and trading platform, one of the most complicated and high-flow online transaction processing (OLTP) systems in the world, according to Von Gosling, senior technologist at Alibaba and RocketMQ co-creator.

The company made it open source in 2012 and donated it to ASF in November 2016. Version 4.0 was released last February.

RocketMQ was designed for queueing, decoupling, buffering and controlling data flow between Alibaba’s many subsystems, Gosling explained. It is high-flow and latency-sensitive for such an online system. RocketMQ put much effort into improving throughput, decreasing response time and eliminating latency spikes.

It’s been widely adopted by thousands of enterprises in China, India and other countries. Clients include Chinese ride-sharing company Didi Chuxing, delivery service S.F. Express, WeBank, Peking University and the Chinese Academy of Sciences.

The distributed messaging engine’s features include:

• Low latency; more than 99.6 percent response latency within 1 millisecond under high pressure;
• Finance-oriented, high availability with tracking and auditing features;
• Industry-sustainable, trillion-level message capacity guaranteed;
• Vendor-neutral, support multiple messaging protocols like JMS and OpenMessaging;
• Big Data-friendly, batch transferring with versatile integration for flooding throughput; and
• Massive accumulation, given sufficient disk space, without performance loss.

The RocketMQ team described the project’s first three stages in an InfoQ post:

1. Push mode in which data are stored in a relational database.
2. Pull mode in which data storage is managed by file systems.
3. Pull mode along with some Push operations. Largely optimized, it has low latency and high performance, especially in high concurrency and massive amounts of data scenarios.

RocketMQ consists of four parts: name servers, brokers, producers and consumers. Name servers provide lightweight service discovery and routing. Brokers take care of message storage by providing lightweight queue mechanisms that group messages by topic. Producers support distributed deployment. Consumers support cluster consumption and message broadcast, providing a real-time message subscription mechanism.

They also compared RocketMQ with other popular messaging platforms, including ZeroMQ, RabbitMQ, ActiveMQ and Kafka.

Developed entirely in China, Gosling described a “cultural collision” when the project entered the Apache Incubator, including language problems and understanding of the Apache Way.

The project has been removing Chinese characters, polishing code style, writing English documents and adding more tests to make the project more user-friendly and to attract to developers around the world, he said.

It’s also been adding features, like sending batch, filtering by SQL92, logappender and implementation of OpenMessaging to make RocketMQ more powerful. Most of them have been released in version 4.1.

To serve the community better, the team created and maintained two repositories, one as the kernel version and the other for community contributions. The community contributed some integrated projects with some other Apache TLPs like Apache Storm, Apache Ignite, Apache Spark and Apache Flume, said Xinyu “yukon” Zhou, member of the Apache RocketMQ Project Management Committee.

OpenMessaging and version 5.0 are on the roadmap. RocketMQ 5.0 will develop more atomic messaging semantics and aim to meet the requirements for complicated cloud-oriented applications and data processing, he said.

It’s also working with other groups within ASF working on projects such as ActiveMQ, RabbitMQ and Kafka to develop an open standard for users to connect these products and relieve the work of changing from one to another.

Feature Image: “Rocket” by Joe Lewis, licensed under CC BY-SA 2.0.