VMware Tanzu RabbitMQ supports Intra-cluster compression where RabbitMQ nodes form clusters and then try to use compression.

This feature is only supported in Tanzu RabbitMQ. It is not supported in open-source RabbitMQ.

RabbitMQ nodes communicate with their peers and CLI tools by using dedicated TCP connections, optionally protected with TLS.

In a heavily loaded system, internode traffic flows can be substantial, approaching or even saturating the bandwidth provided by network links. Compression of this traffic helps reduce the load on bandwidth available, in some cases by more than 50% depending on the nature of the workload.

How to use intra-cluster compression

Intra-cluster compression is turned off by default in Tanzu RabbitMQ. To turn on intra-cluster compression, set the environment variable RABBITMQ_SERVER_START_ARGS="-proto_dist inet_tcp_compress". When RabbitMQ nodes that have this environment variable set form a cluster, these nodes try to use compression.

For the data to be compressed, the following conditions must be met:

• Both RabbitMQ nodes must support intra-cluster compression. In other words, both nodes must run Tanzu RabbitMQ.

• Both nodes must share at least one compression algorithm in common.

When the conditions are not met, nodes communicate as they otherwise do without traffic compression. This means that Tanzu RabbitMQ remains compatible with open-source RabbitMQ. It is therefore possible to switch from one RabbitMQ edition to the other without stopping the entire cluster.

How intra-cluster compression works

The first time a RabbitMQ node tries to contact another node, it does the following:

• After the TCP connection is open, compression is not used at first.

• The node initiating the connection detects if the remote peer has Erlang distribution compression support. If it does not, the connection remains uncompressed and following steps are skipped.

• After the node is sure the remote node supports compression, it negotiates the compression algorithm to use. To start this process, it sends a message to the remote node and specifies the list of algorithms it supports.

• The remote node compares the received list of algorithms to its own list. The remote node’s list is ordered by preference. The selected algorithm is the first one in the remote node’s list that is also supported by the initiating node. If there is no algorithm in common, the connection remains uncompressed and the following steps are skipped.

• After an algorithm is selected, the remote node sends a message back to the initiating node to inform it of its decision.

• The two nodes sync to start compression on the existing TCP connection.

How to get a license

For more information or to get a quote, visit the Tanzu RabbitMQ product page.

Limitations

• Intra-cluster compression and TLS cannot be used at the same time. Currently they are mutually exclusive.

  This is because a RabbitMQ node is configured with a specific distribution module, provided with Tanzu RabbitMQ, and a small add-on to deal with the algorithm negotiation. The distribution module is a replacement for the default module (inet_tcp_dist) or the TLS-enabled module (inet_tls_dist). It is impossible to use two modules simultaneously.

• In Tanzu RabbitMQ, the compression code relies on native libraries. Only Linux/amd64 is supported in the existing packaging. More platforms might be compiled for in the future.