Tanzu RabbitMQ on Kubernetes 4.0

4.0 3.13

Warm Standby Replication

Last Updated February 04, 2025

Tanzu RabbitMQ for Kubernetes supports Warm Standby Replication (WSR), a strategy which replicates or copies data (continuous schema definitions and messages) from an upstream (primary) RabbitMQ cluster to a downstream (standby) cluster.

If the upstream (primary) cluster fails then an administrator can quickly start the recovery process, which means putting the downstream (standby) cluster into service as the new upstream (primary) cluster with minimal downtime or data loss. This information describes configuring WSR when using the Kubernetes operators.

This feature is only supported in Tanzu RabbitMQ. It is not supported in the open-source RabbitMQ product.

Learn the terminology

  • Upstream Cluster:

    The primary active cluster is most often called the “upstream” cluster. The “upstream” cluster reference is used in the configuration files, which you come across when you configure the WSR feature. For the rest of this topic, it is referred to as the upstream (primary) cluster.

  • Downstream Cluster:

    The standby remote cluster is most often called the “downstream” cluster. The “downstream” cluster reference is used in the configuration files, which you come across when you configure the WSR feature. For the rest of this topic, it is referred to as the downstream (standby) cluster.

  • Schema:

    Nodes and clusters store information that can be referred to as a schema, metadata, or topology. Users, virtual hosts, queues, exchanges, bindings, and runtime parameters are all in this category. This metadata is called definitions in RabbitMQ.

  • Sync Request:

    A sync request carries a payload that enables the upstream (primary) side to compute the difference between the schemas on the upstream (primary) and downstream (standby) clusters.

  • Sync Response:

    A sync response carries the difference plus all the definitions that are only present on the upstream (primary) side or conflict. The downstream (standby) side uses this information to apply the definitions. Any entities only present on the downstream are deleted, which ensures that downstreams follow their upstream’s schema as closely as possible.

  • Sync Operation:

    A sync operation is a request/response sequence that involves a sync request sent by the downstream (standby) cluster and a sync response that the upstream (primary) cluster sends back.

  • Schema Replication:

    This is the automated process of continually replicating schema definitions from an upstream (primary) cluster to one or more downstream (standby) clusters.

  • Message Replication:

    This is the automated process of continually replicating published messages from an upstream (primary) cluster to one or more downstream (standby) clusters.

  • Loose Coupling:

    The upstream and its followers (downstreams) are loosely connected. If one end of the schema replication connection fails, the delta between clusters’ schema grows but neither is affected in any other way. This applies to the message replication connection as well. If an upstream is under too much load to serve a definition request, or the sync plug-in is unintentionally disabled, the downstream does not receive responses for sync requests for a period of time.

    (This applies to the message replication connection as well.) If an upstream is under too much load to serve a definition request, or the sync plugin is unintentionally disabled, the downstream won’t receive responses for sync requests for a period of time. If a downstream fails to apply definitions, the upstream is not affected and neither are its downstream peers.

    Therefore, availability of both sides are independent of each other. When multiple downstreams are syncing from a shared upstream, they do not interfere or coordinate with each other. Both sides have to do a little more work. On the upstream side, this load is shared between all cluster nodes. On the downstream side, the load should be minimal in practice, assuming that sync operations are applied successfully so that the delta does not accumulate.

  • Downstream Promotion:

    Promoting the downstream (standby) cluster to the upstream (primary) cluster.

Why use Warm Standby Replication

With the Warm Standby Replication feature, you get:

  • A disaster recovery strategy that continually replicates schema definitions and automatically replicates enqueued messages from the upstream (primary) cluster to the downstream (standby) cluster. If a disaster occurs on the upstream (primary) cluster, an administrator can quickly start the recovery process with minimal downtime or data loss.

  • A way of transferring schema definitions in a compressed binary format that reduces bandwidth use.

  • Cluster co-dependencies are avoided because all communication between the sides is completely asynchronous. For example, a downstream (standby) cluster can run a different version of RabbitMQ.

  • Links to other clusters are easy to configure, which is important for disaster recovery. For example, if you are setting up more than one downstream (standby) cluster.

What is replicated?

Replicated

  • Schema definitions such as virtual hosts, queues, users, exchanges, bindings, runtime parameters, and so on.
  • Messages that are published to quorum queues, classic queues, and streams.

Not Replicated

Schema synchronization does not synchronize Kubernetes objects.

How Warm Standby Replication works

The WSR process uses plug-ins described in the following sections.

Continuous Schema Replication (SchemaReplication) plug-in

The Continuous Schema Replication plug-in connects the upstream (primary) cluster to the downstream (standby) cluster with a schema replication link. The downstream (standby) clusters connect to their upstream (primary) cluster and initiate sync operations. These operations synchronize the schema definition on the downstream side with the same schema definition of that on the upstream side.

A node running in the downstream mode (a follower) can be converted to an upstream (leader) on the fly. This makes the node disconnect from its original source, therefore stopping all syncing. The node then continues operating as a member of an independent cluster, no longer associated with its original upstream. Such conversion is called a downstream promotion and should be completed in case of a disaster recovery event.

Standby Message Replication plug-in

To ensure improved data safety and reduce the risk of data loss, it is not enough to automate the replication of RabbitMQ entities (schema objects). The Warm Standby Replication feature implements a hybrid replication model.

In addition to schema definitions, it also manages the automated and continuous replication of enqueued messages from the upstream (primary) cluster. During the setup process, a replication policy is configured at the vhost level in the upstream (primary) cluster indicating the downstream queues that should be matched and targeted for message replication.

Messages and relevant metrics from the upstream queues are then pushed to the downstream queues via a streaming log which the downstream(s) subscribe to. Currently, quorum queues, classic queues, and stream queues are supported for message replication.

To ensure improved data safety and reduce the risk of data loss, it is not enough to automate the replication of RabbitMQ entities (schema objects). The WSR feature implements a hybrid replication model. In addition to schema definitions, it manages the automated and continuous replication of enqueued messages from the upstream (primary) cluster.

During the setup process, a replication policy is configured at the virtual host level in the upstream (primary) cluster, indicating the downstream queues that should be matched and targeted for message replication. Messages and relevant metrics from the upstream queues are then pushed to the downstream queues by using a streaming log that the downstream(s) subscribe to. Currently, quorum queues, classic queues, and stream queues are supported for message replication.

For quorum queues and classic queues, RabbitMQ clusters replicate messages in the queues to the downstream (standby) cluster, but these messages are not published in the queues in the downstream (standby) cluster until that downstream (standby) cluster is promoted to the upstream (primary) cluster.

Every 60 seconds (by default) the timestamp of the oldest message in each queue is sent to the downstream (standby) cluster. The promotion process uses the timestamp as a cutoff point for message recovery.

So, based on the timestamp of the oldest message, all messages from the oldest timestamp to the current timestamp are recovered when the promotion process happens. With this process, duplicate messages are likely to happen in a busy RabbitMQ cluster scenario because of the timestamp refresh interval and stale messages pushing the timestamp further into the past.

However, there is also the guarantee of not losing or missing any messages during the recovery process. Streams work differently. Messages are replicated directly to streams on the downstream (standby) cluster. However, you cannot publish new messages to streams on the downstream (standby) cluster until the downstream (standby) cluster is promoted to the upstream (primary) cluster.

Requirements for Warm Standby Replication

Requirements for WSR are:

  • The Continuous Schema Replication (SchemaReplication) and Standby Message Replication (StandbyReplication) plug-ins must be enabled. See if the plug-ins are enabled by running:

    rabbitmq-plugins list rabbitmq_schema_definition_sync
rabbitmq-plugins list rabbitmq_standby_replication

    The following is an example of the output that is returned when the Continuous Schema Replication (SchemaReplication) plug-in is enabled:

    rabbitmq [ ~ ]$ rabbitmq-plugins list rabbitmq_schema_definition_sync
Listing plugins with pattern "rabbitmq_schema_definition_sync" ...
Configured: E = explicitly enabled; e = implicitly enabled
| Status: * = running on rabbit@6b4e8ac05412
|/
[E*] rabbitmq_schema_definition_sync

    You can enable the Continuous Schema Replication (SchemaReplication) and Standby Message Replication (StandbyReplication) plug-ins at the same time by running:

    rabbitmqctl enable_warm_standby

  • You must know the credentials (user name and password) that you want to use for WSR.

Set up and configure Warm Standby Replication

There can be multiple downstream (standby) clusters linked to one upstream (primary) cluster. This setup describes one upstream cluster and one downstream cluster.

WSR uses stream queues to log/copy changes. As a result, the number of messages can grow to be very large, but because RabbitMQ streams can store large amounts of data efficiently, minimal memory is used.

Set up and configure the upstream (primary) RabbitMQ cluster

The YAML in the following steps is just an example. Update the YAML as required for your environment.

If it is not specified, always set the cluster size (initial-cluster-size) to at least 3.

  1. Set up and configure the upstream (primary) RabbitMQ cluster with the required plug-ins Continuous Schema Replication (SchemaReplication) and Standby Message Replication, the operating mode, the endpoints, and so on. The schema_definition_sync.operating_mode field must be set to upstream to provide upstream-related configurations.

    The schema_definition_sync.connection.endpoints.one, which is set to 5672, is the service external IP of the upstream (primary) cluster. If you are securing WSR with TLS, as mentioned later in the topic, you must change the port 5672 in the following example to 5671.

    apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: upstream-rabbit
spec:
  ...
  rabbitmq:
    additionalPlugins:
      - rabbitmq_stream
      - rabbitmq_stream_management
      - rabbitmq_schema_definition_sync
      - rabbitmq_schema_definition_sync_prometheus  # optional
      - rabbitmq_standby_replication
      - rabbitmq_warm_standby
    additionalConfig: |
      log.console.level = debug
      schema_definition_sync.operating_mode = upstream
      schema_definition_sync.connection.endpoints.one = localhost:5672
      schema_definition_sync.connection.username = test-user
      schema_definition_sync.connection.password = test-password
      standby.replication.operating_mode = upstream
      standby.replication.connection.endpoints.one = localhost:5552
      standby.replication.connection.username = test-user
      standby.replication.connection.password = test-password
      # message stream retention limit (can either be size or time based)
      standby.replication.retention.size_limit.messages = 5000000000
      # standby.replication.retention.time_limit.messages = 12h

  2. Verify the status of the upstream (primary) RabbitMQ cluster by running:

    kubectl get pod -n NAMESPACE

    Where NAMESPACE is your namespace. For example,rbtmq-cluster.

    Verify that the output looks similar to this example:

    NAME READY STATUS RESTARTS AGE
upstream-rabbit-server-0 1/1 Running 1 28d
upstream-rabbit-server-1 1/1 Running 1 28d
upstream-rabbit-server-2 1/1 Running 1 28d

    You can also verify that the upstream (primary) services were created properly by running:

    kubectl get svc -n NAMESPACE

    Where NAMESPACE is your namespace. For example,rbtmq-cluster.

    Example output:

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
upstream-rabbit NodePort 10.100.153.24 <none> 5672:32516/TCP,15672:31792/TCP,5552:30702/TCP,15692:31009/TCP 28d
upstream-rabbit-nodes ClusterIP None <none> 4369/TCP,25672/TCP 28d

  3. Configure a secret to contain a replication-schema user and user credentials. This user is used from the downstream (standby) cluster to establish a connection and manage the replication.

    apiVersion: v1
kind: Secret
metadata:
  name: upstream-secret
type: Opaque
stringData:
  username: test-user
  password: test-password
---
apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
  name: rabbitmq-replicator
spec:
  rabbitmqClusterReference:
    name:  upstream-rabbit # the upstream RabbitMQ cluster name. It must be in the same namespace and it is a mandatory value.
  importCredentialsSecret:
    name: upstream-secret

  4. Add the write, configure, and read permissions for the user on the rabbitmq_schema_definition_sync virtual host. These permissions are required to ensure that the Continuous Schema Replication (SchemaReplication) plug-in operates correctly. The following YAML code provides an example of how to configure these permissions on the rabbitmq_schema_definition_sync virtual host.

    The rabbitmq_schema_definition_sync virtual host is created when the cluster is created. It is this virtual host that is referenced in the following code, therefore for this example this virtual host is replicated.

    apiVersion: rabbitmq.com/v1beta1
kind: Permission
metadata:
  name: rabbitmq-replicator.rabbitmq-schema-definition-sync.all
spec:
  vhost: "rabbitmq_schema_definition_sync" # name of a virtual host
  userReference:
    name: rabbitmq-replicator
  permissions:
    write: ".*"
    configure: ".*"
    read: ".*"
  rabbitmqClusterReference:
    name: upstream-rabbit  # the upstream RabbitMQ cluster name. It must be in the same namespace and it is a mandatory value.

    If you want to replicate additional virtual hosts, you must create them and add the write, configure, and read permissions to them. For example, if you want a test virtual host to be part of the replication process, create it (adding the tag standby_replication) in this way. The standby_replication tag and permissions are used by the plug-in to select the virtual host to replicate.

    apiVersion: rabbitmq.com/v1beta1
kind: Vhost
metadata:
  name: default
spec:
  name: "test" # vhost name
  tags: ["standby_replication"]
  rabbitmqClusterReference:
    name: upstream-rabbit

    Next, apply the write, configure, and read permissions:

    apiVersion: rabbitmq.com/v1beta1
kind: Permission
metadata:
  name: rabbitmq-replicator.rabbitmq-schema-definition-sync.test
spec:
  vhost: "test" # name of the vhost
  userReference:
    name: rabbitmq-replicator
  permissions:
    write: ".*"
    configure: ".*"
    read: ".*"
  rabbitmqClusterReference:
    name: upstream-rabbit  # the upstream RabbitMQ cluster name. It must be in the same namespace and it is a mandatory value.

  5. Add the replication policies. A replication policy indicates the RabbitMQ downstream objects, such as queues or exchanges, that should be matched and targeted for message replication in the virtual host that was previously created:

    apiVersion: rabbitmq.com/v1beta1
kind: Policy
metadata:
  name: upstream-policy
spec:
  name: osr # name of the policy
  vhost: "test" # default to '/' if not provided
  pattern: "^.*" # regex used to match queues and exchanges
  applyTo: "queues" # set to 'queues', 'exchanges', or 'all'
  definition: # policy definition
    remote-dc-replicate: true
  rabbitmqClusterReference:
    name: upstream-rabbit # rabbitmqCluster must exist in the same namespace as this resource

Set up and configure the downstream (standby) RabbitMQ cluster

For the downstream (standby) cluster, the optional sync operation interval setting schema_definition_sync.downstream.minimum_sync_interval can be configured. It is included in the example later in this section. This interval is in seconds and controls how often the downstream (standby) node initiates the schema to be synchronized.

For example, the setting schema_definition_sync.downstream.minimum_sync_interval = 30 initiates sync operations every 30 seconds. There is no interval for message synchronization because messages are synchronized continuously.

The downstream (standby) node can be configured to exclude certain entities (queues, exchanges, users, and so on) from the synchronization process. For example, you can filter the user local-admin from the synchronization process.

This can be useful if you want to include certain entities in the downstream (standby), even if they do not exist in the upstream (primary). The downstream (standby) synchronization process deletes any entity not present in the upstream (primary).

To set up and configure the downstream (standby) RabbitMQ cluster:

  1. Set up and configure the downstream (standby) cluster with the required plug-ins Continuous Schema Replication (SchemaReplication) and Standby Message Replication, the operating mode, the endpoints, and so on, as in this example:

    # to filter users using regex
schema_definition_sync.downstream.locals.users = ^my-user$

# to filter vhosts using regex
schema_definition_sync.downstream.locals.vhosts = ^vhost-test.*

# to filter policies using regex
schema_definition_sync.downstream.locals.policies = ^example$

# to filter global parameters
schema_definition_sync.downstream.locals.global_parameters = ^some-param

# to filter parameters
schema_definition_sync.downstream.locals.parameters = example$

# to filter queues
schema_definition_sync.downstream.locals.queues = ^leave-this-q$

# to filter exchanges
schema_definition_sync.downstream.locals.exchanges = ^important-exchange-[a-z]+

    The schema_definition_sync.operating_mode field must be set downstream to provide downstream related configurations.

    schema_definition_sync.connection.endpoints.one, which is set to 5672, is the service external IP of the upstream (primary) cluster. If you are securing WSR with TLS, as mentioned later later in this topic, you must change the port to 5671. It is 5672 in this example.

    standby.replication.connection.endpoints.one is the endpoint to connect to the upstream RabbitMQ cluster. Endpoints must be reachable from this downstream cluster with the stream protocol port. If you are securing WSR with TLS, the stream protocol port must be changed to 5551.

    The following YAML is just an example. Edit it as required for your environment.

    apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: downstream-rabbit
spec:
  ...
  rabbitmq:
    additionalPlugins:
      - rabbitmq_stream
      - rabbitmq_stream_management
      - rabbitmq_schema_definition_sync
      - rabbitmq_schema_definition_sync_prometheus   # optional
      - rabbitmq_standby_replication
      - rabbitmq_warm_standby
    additionalConfig: |
      schema_definition_sync.operating_mode = downstream
     # initiate sync operations every 30 seconds
      schema_definition_sync.downstream.minimum_sync_interval = 30

      schema_definition_sync.connection.endpoints.one = upstream-rabbit:5672
      schema_definition_sync.connection.username = test-user
      schema_definition_sync.connection.password = test-password

      schema_definition_sync.downstream.locals.users = ^default_user_
      # schema_definition_sync.downstream.locals.permissions = ^default_user_
      schema_definition_sync.downstream.locals.queues = ^local
      schema_definition_sync.downstream.locals.parameters = ^standby
      schema_definition_sync.downstream.locals.global_parameters = ^standby

      standby.replication.operating_mode = downstream

      standby.replication.connection.endpoints.one = upstream-rabbit:5552
      standby.replication.connection.username = test-user
      standby.replication.connection.password = test-password
     # message stream retention limit (can either be size or time based)
      standby.replication.retention.size_limit.messages = 5000000000
     # standby.replication.retention.time_limit.messages = 12h

    The upstream-rabbit value in the schema_definition_sync.connection.endpoints.one and standby.replication.connection.endpoints.one parameters is the DNS or external IP of the upstream (primary) cluster.

  2. See the status of the downstream (standby) RabbitMQ cluster by running:

    kubectl get pod -n NAMESPACE

    Where NAMESPACE is your namespace. For example, rbtmq-cluster.

    Verify that the output is similar to this example:

    NAME READY STATUS RESTARTS AGE
downstream-rabbit-server-0 1/1 Running 1 28d
downstream-rabbit-server-1 1/1 Running 1 28d
downstream-rabbit-server-2 1/1 Running 1 28d

    You can also verify that the downstream (standby) services are created properly by running:

    kubectl get svc -n NAMESPACE

    Where NAMESPACE is your namespace. For example, rbtmq-cluster.

    Verify that the output is similar to this example:

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
downstream-rabbit NodePort 10.100.198.3 <none> 5672:31414/TCP,15672:30479/TCP,5552:30399/TCP,15692:32563/TCP 28d
downstream-rabbit-nodes ClusterIP None <none> 4369/TCP,25672/TCP 28d

Update the replication configuration

Important Notes

  • Update operations are supported, such as updating the policy name, virtual host, patterns, and adding a new policy.

  • Updating the replication policies is not fully supported. When you remove the definition of an existing policy from the YAML file, the existing policy is not removed when you update the configuration. The policy still exists in the RabbitMQ definition. However, if you add a new policy to the YAML file, it creates the new policy in RabbitMQ.

  • The ...operating_mode and rabbitmqClusterReference fields cannot be changed. If you need to update these fields, then you must delete the Warm Standby Replication custom resources on the upstream (primary) and downstream (standby) clusters, and then complete the previous sections again to configure them.

(Optional) Configure Warm Standby Replication with TLS

To configure the upstream (primary) and downstream (standby) clusters to complete replication over TLS, which secures communications between the clusters:

  1. Configure your clusters with secrets containing TLS certificates by following this TLS example.

  2. You can then use these certificates by including the ssl_options configuration parameters in the configuration file. Include these parameters in the same format as the ssl_options, which are detailed in Enabling TLS Support in RabbitMQ.

  3. Configure your clusters with secrets containing TLS certificates by following this TLS example.

  4. On the upstream cluster, set the parameters under schema_definition_sync.ssl_options:

    apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: upstream-rabbit
spec:
...
  tls:
    secretName: tls-secret
  rabbitmq:
    additionalPlugins:
    - rabbitmq_stream
    - rabbitmq_schema_definition_sync
    - rabbitmq_schema_definition_sync_prometheus
    - rabbitmq_standby_replication
    additionalConfig: |
     log.console.level = debug
       schema_definition_sync.operating_mode = upstream
       standby.replication.operating_mode = upstream
       standby.replication.retention.size_limit.messages = 5000000000
       schema_definition_sync.ssl_options.certfile              = /etc/rabbitmq-tls/tls.crt
       schema_definition_sync.ssl_options.keyfile               = /etc/rabbitmq-tls/tls.key
       schema_definition_sync.ssl_options.verify                = verify_none
       schema_definition_sync.ssl_options.fail_if_no_peer_cert  = false
       schema_definition_sync.connection.username = test-user
       schema_definition_sync.connection.password = test-password
       standby.replication.connection.endpoints.one = localhost:5552
       standby.replication.connection.username = test-user
       standby.replication.connection.password = test-password
    # message stream retention limit (can either be size or time based)
       standby.replication.retention.size_limit.messages = 5000000000
    # standby.replication.retention.time_limit.messages = 12h

  5. On the downstream cluster, set the parameters under schema_definition_sync.ssl_options and standby.replication.downstream.ssl_options:

    apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: downstream-rabbit
spec:
...
  tls:
    secretName: tls-secret
  rabbitmq:
    additionalPlugins:
    - rabbitmq_stream
    - rabbitmq_schema_definition_sync
    - rabbitmq_schema_definition_sync_prometheus
    - rabbitmq_standby_replication
    additionalConfig: |
       schema_definition_sync.operating_mode = downstream
       standby.replication.operating_mode = downstream
       schema_definition_sync.downstream.locals.users = ^default_user_
       schema_definition_sync.downstream.locals.global_parameters = ^standby
       standby.replication.retention.size_limit.messages = 5000000000
       schema_definition_sync.ssl_options.certfile = /etc/rabbitmq-tls/tls.crt
       schema_definition_sync.ssl_options.keyfile = /etc/rabbitmq-tls/tls.key
       schema_definition_sync.ssl_options.verify = verify_none
       schema_definition_sync.ssl_options.fail_if_no_peer_cert = false
       standby.replication.downstream.ssl_options.certfile = /etc/rabbitmq-tls/tls.crt
       standby.replication.downstream.ssl_options.keyfile = /etc/rabbitmq-tls/tls.key
       standby.replication.downstream.ssl_options.verify = verify_none
       standby.replication.downstream.ssl_options.fail_if_no_peer_cert = false
       schema_definition_sync.connection.username = test-user
       schema_definition_sync.connection.password = test-password
       schema_definition_sync.connection.endpoints.one = upstream-rabbit:5672
       standby.replication.connection.endpoints.one = upstream:5552
       standby.replication.connection.username = test-user
       standby.replication.connection.password = test-password
    # message stream retention limit (can either be size or time based)
       standby.replication.retention.size_limit.messages = 5000000000
    # standby.replication.retention.time_limit.messages = 12h

    Peer verification (normally configured by setting ssl_options.verify to verify_peer) is not supported for Warm Standby Replication. schema_definition_sync.ssl_options.verify and standby.replication.downstream.ssl_options.verify must be set to verify_none.

(Optional) Using Vault for secrets

Operator supports fetching RabbitMQ credentials from HashiCorp Vault. If you use Vault to store your RabbitMQ user credentials, you can provide a path to read credentials from Vault instead of providing a Kubernetes secret. The credentials must have been written to Vault already before referencing them in custom resources. The Vault secret must have fields username and password.

The following example shows how to configure the SchemaReplication custom resource to get credentials from Vault:

---
apiVersion: rabbitmq.com/v1beta1
kind: SchemaReplication
metadata:
  name: downstream
spec:
  secretBackend:
    vault:
      secretPath: path/to/rabbitmq/creds # instead of spec.upstreamSecret
  ...

The following example shows how to configure the StandbyReplication custom resource to get credentials from Vault:

---
apiVersion: rabbitmq.tanzu.vmware.com/v1beta1
kind: StandbyReplication
metadata:
  name: downstream
spec:
  operatingMode: "downstream"
  downstreamModeConfiguration:
    secretBackend:
      vault:
        secretPath: path/to/rabbitmq/creds # instead of spec.downstreamModeConfiguration.upstreamSecret
  ...

(Optional) Configure schema replication for shovel and federation plug-ins

You can configure the upstream (primary) cluster to replicate shovel and federation runtime parameters on the downstream (standby) cluster. You must opt in and select the virtual hosts to replicate the runtime parameters.

The following is an example of the configuration parameters you must add to additionalConfig to enable schema replication of both shovel and federation plug-ins:

schema_definition_sync.opt_in.sync_federation = true
schema_definition_sync.opt_in.sync_shovels = true

schema_definition_sync.opt_in.shovel_selector = ^upstream_shovels_parameter
schema_definition_sync.opt_in.federation_selector = ^upstream_federation_parameter

This can be a tricky operation because the downstream (standby) cluster connects to where production connects, which could be a remote location. For this reason, it is important to be able to connect and disconnect shovel/federation during runtime by running:

rabbitmqctl enable_shovel_schema_replication
rabbitmqctl disable_shovel_schema_replication
rabbitmqctl enable_federation_schema_replication
rabbitmqctl disable_federation_schema_replication

Verify Warm Standby Replication is configured correctly

To verify that WSR is configured correctly:

  1. See the status of the Continuous Schema Replication (SchemaReplication) and Standby Message Replication (StandbyReplication) plug-ins by running:

    rabbitmqctl schema_replication_status
rabbitmqctl standby_replication_status

  2. Check if the topology objects are replicated in the downstream (standby) RabbitMQ cluster. You can do this by logging in to the RabbitMQ management UI for that specific cluster or by running the rabbitmqctl command with kubectl exec from the command line.

    After using either method, your upstream RabbitMQ topology (virtual hosts, users, queues, exchanges, policies, and so) are returned and listed in the downstream cluster. If you do not see these topology objects, you can check the upstream (primary) and downstream (standby) RabbitMQ clusters’ logs to investigate the issue.

  3. Pod exec to downstream RabbitMQ pods one by one and run the following command to list the virtual hosts with local data to recover. This list contains all virtual hosts that you tagged with standby_replication:

    rabbitmqctl list_vhosts_available_for_standby_replication_recovery

  4. If you published messages to classic queues or quorum queues in the virtual hosts that are tagged with standby_replication, and covered by replication policies (as mentioned earlier in the topic), you can list the number of messages replicated for each virtual host, exchange, and routing key. This operation returns the number of replicated messages that can be published to a specific exchange with a specific routing key.

    If the exchange is empty or missing, it means that the message was published to the default exchange. The routing key might not be the same as the name of the queue.

    It does not return messages per queue. The routing of messages to their destination queue and the consumption of these messages does happen until promotion is initiated, which must be taken into consideration when you are interpreting these numbers.

    If you set up replication recently then the number of available messages should be small, meaning that this this operation should run quickly. If the amount of available data is substantial, this operation can take longer.

    • Method 1

      1. Access the RabbitMQ Management UI to see the replication information. A link to the RabbitMQ management interface is on the details page for your hosted RabbitMQ solution. If you have RabbitMQ installed on localhost, go to http://localhost:15672/ to find the management page.

      2. Select the Admin tab. You can then access the Schema Replication, Standby Replication, and Standby Promotion tabs in the Admin section.

      VMware RabbitMQ Management UI for Warm Standby Replication.

    • Method 2

      Run the following command:

      rabbitmq-diagnostics inspect_local_data_available_for_standby_replication_recovery

  5. If you published messages to streams in the virtual hosts that are tagged with standby_replication, and covered by replication policies (mentioned earlier in the topic), you can list the number of messages replicated for each virtual host and stream. Again, you can do this either by logging in to the RabbitMQ Management UI and following the instructions in the previous Method 1 or by running:

    rabbitmq-diagnostics inspect_local_stream_data_available_for_standby_replication_recovery

    If you just want the list of streams available for Standby Replication recovery, run:

    rabbitmqctl list_streams_available_for_standby_replication_recovery [--vhost <vhost>]

Promote the downstream (standby) cluster for disaster recovery

A downstream (standby) cluster with synchronized schema and messages is only useful if it can be turned into a new upstream (primary) cluster in case of a disaster event. This process is known as downstream promotion.

Promotion and reconfiguration happen on the fly and do not involve RabbitMQ node restarts or redeployment.

In the case of a disaster event, the recovery process involves several steps:

  • The downstream (standby) cluster is promoted to an upstream (primary) cluster by the service operator. When this happens, all upstream links are closed and for every virtual host, unacknowledged messages are re-published to their original destination queues. This just applies to messages belonging to classic queues and quorum queues. Streams are already stored in the target stream on the downstream (standby) cluster.

  • Applications are redeployed or reconfigured to connect to the newly promoted upstream (primary) cluster.

  • Other downstream (standby) clusters must be reconfigured to follow the new promoted cluster.

When downstream promotion happens, a promoted downstream (standby) cluster is detached from its original upstream (primary) cluster. It then operates as an independent cluster that can be used as an upstream (primary) cluster. It does not sync from its original upstream but can be configured to collect messages for offsite replication to another data center.

The downstream promotion process takes time. The amount of time it takes is proportional to the retention period used. This operation is only intensive for CPU and disk I/O when queues are used. It is not for streams because streams are just restarted. Messages are already stored in the target streams.

Every downstream node is responsible for recovering the virtual hosts it owns, which helps distribute the load between cluster members. To list virtual hosts available for downstream promotion (in other words, that have local data to recover), run:

rabbitmqctl list_vhosts_available_for_standby_replication_recovery

The easiest way to promote a downstream (standby) cluster is to use the HTTP API. The API does not accept any arguments. It is a HTTP POST request to http://<i>{node-hostname}</i>:15672/api/tanzu/osr/downstream/promote.

Other ways to complete the promotion process include running one of the following commands. If you have existing deployments that use the command

rabbitmqctl promote_standby_replication_downstream_cluster

then continue using this command, and then complete the steps in the following Post Promotion section.

Alternatively, if you have new deployments, you can run the command

rabbitmqctl promote_warm_standby

to finish all promotion tasks in one step. You do not need to finish the post promotion steps after running this command, apart from the step to erase the replicated data on the old downstream (standby) cluster. However, finishing this step is optional.

The arguments --start-from-scratch, --all-available, and --exclude-virtual-hosts are applicable to both commands for quorum queues and classic queues.

To promote a downstream (standby) cluster (in other words, to start the disaster recovery process), run:

rabbitmqctl promote_standby_replication_downstream_cluster [--start-from-scratch][--all-available] \
[--exclude-virtual-hosts \"<vhost1>,<vhost2>,<...>\"]

Alternatively, you can finish the promotion by running:

rabbitmqctl promote_warm_standby
[--start-from-scratch][--all-available] [--exclude-virtual-hosts \"<vhost1>,<vhost2>,<...>\"]

Where:

  • --start-from-scratch recovers messages from the earliest available data instead of the last timestamp recovered previously, even if information about the last recovery is available.
  • --all-available forces the recovery of all messages that are available if neither the last cutoff nor the last recovery information is available.
  • --exclude-virtual-hosts excludes virtual hosts from promotion.

The arguments --start-from-scratch, --all-available, and --exclude-virtual-hosts do not apply to streams because messages are replicated automatically to the streams on the downstream cluster. So, if you replicate from streams only, to finish promotion you only need to run:

rabbitmqctl promote_standby_replication_downstream_cluster

To display the promotion summary in case a promotion was attempted, run:

rabbitmqctl display_standby_promotion_summary

The recovery process stores a summary on disk, indicating the last timestamp that was recovered. Earlier messages are skipped when there are subsequent recoveries, which prevents duplicate recovered messages. The --start-from-scratch option skips this check, and always recovers everything.

During promotion of the downstream cluster, WSR does not support the recovery of messages that are routed to target queues by the AMQP v0.9.1 BCC header.

After the recovery process finishes, the cluster can be used as usual.

Post promotion

After promoting the downstream (standby) cluster to be the upstream (primary) cluster for disaster recovery, do these steps if necessary:

  1. If you need to restart the promoted cluster, change the operatingMode: "downstream" to operatingMode: "upstream" because this modification does not happen automatically when the cluster is restarted.

    If you don’t change it, the promoted downstream cluster (which is now the upstream cluster) runs in the downstream mode because it is still a downstream cluster in its definition file.

    The original upstream (primary) cluster that experienced a disaster event can be:

    • Brought back as a downstream (standby) cluster for the newly promoted upstream (primary) cluster
    • Promoted back as the upstream (primary) cluster
    • Not used at all

  2. After promotion, you can erase the replicated data on the old downstream (which is effectively the new promoted upstream) from the disk.

    For example, a cluster in Dublin is the upstream (primary) cluster and a cluster in London is the downstream (standby) cluster. The cluster in London gets promoted to be the upstream (primary) cluster. After promotion, you can now remove previous downstream-related data from the cluster in London (because it is now promoted and running as the upstream cluster) by running:

    rabbitmqctl delete_all_data_on_standby_replication_cluster

Running diagnostics

When using classic queues or quorum queues, you can inspect the number of messages replicated for each virtual host, exchange, and routing key from the RabbitMQ Management UI. For more information, see Method 1 in the Verify Warm Standby Replication is configured correctly section.

Running diagnostics takes a long time because it reads and parses all data on disk. This operation can take a substantial time to run even for medium data sizes.

Alternatively, you can access the same information by running:

rabbitmq-diagnostics inspect_local_data_available_for_standby_replication_recovery

The same applies to streams. To access replication information from the RabbitMQ Management UI, see Method 1 in the Verify Warm Standby Replication is configured correctly section or simply run:

rabbitmq-diagnostics inspect_local_stream_data_available_for_standby_replication_recovery

Other useful replication commands

Other useful replication commands include:

  • Start, stop, or restart the replication in both the Schema Replication (SchemaReplication) and Standby Replication (StandbyReplication) plug-ins at the same time by running:

    rabbitmqctl enable_warm_standby
rabbitmqctl disable_warm_standby
rabbitmqctl restart_warm_standby

  • If the cluster size changes, the virtual hosts owned by every node might change. Delete the data for the virtual hosts that nodes no longer own by running:

    rabbitmqctl delete_orphaned_data_on_standby_replication_downstream_cluster

Troubleshooting Warm Standby Replication

Learn how to isolate and resolve problems with WSR in the following sections.

Pod termination takes a long time

Problem

Example: The prestop hook runs

rabbitmq-upgrade await_online_quorum_plus_one -t 604800

which causes the Pod termination to stop until this command succeeds.

Solution

In a 3-node cluster, RabbitMQ must have two nodes available and all queues/streams must be in sync to safely shutdown a Pod. If any queue does not have sufficient in-sync replicas, it is not safe to delete the pod. Inspect the RabbitMQ Management UI to identify which queues do not have enough available synchronized replicas.

If the number of total replicas is not equal to the cluster size (initial-cluster-size), or 3 if the cluster size is greater than 3, then grow the quorum queue or add replicas to a stream. In the example of a 3-node cluster, consider growing/adding replicas to any quorum queue/stream that has fewer than 3 replicas.

To prevent this problem occurring in the future, set the cluster size (initial-cluster-size) to at least 3 at all times.

Messages are collected in the upstream (primary) cluster but are not delivered to the downstream (standby) cluster

Message transfer for quorum queues and classic queues differs from that of streams. The following information describes both replication strategies independently.

Quorum queue and classic queue messages and message acknowledgements are continually stored in the upstream (primary) cluster. The downstream (standby) cluster connects to the upstream (primary) cluster.

The downstream (standby) cluster reads from the internal stream on the upstream (primary) cluster where the messages are stored, and then stores these messages in an internal stream log in the downstream (standby) cluster. These messages are not visible in the RabbitMQ Management UI until promotion, and they must be inspected by running the following command in the downstream:

rabbitmq-diagnostics inspect_local_data_available_for_standby_replication_recovery
# Inspecting local data replicated for multi-DC recovery
# exchange  messages  routing_key  vhost
# myexchange  2  demo  /
#   2  ha.qq  /

To inspect the information about the queue metrics stored in the downstream (standby) cluster, run:

rabbitmq-diagnostics inspect_standby_downstream_metrics
# Inspecting standby downstream metrics related to recovery...
# queue  timestamp  vhost
# ha.qq  1668785252768  /

If the previous command returns the name of a queue called example, it means that the downstream (standby) cluster has messages for queue example ready to be re-published in the event of promoting the downstream (standby) cluster for disaster recovery.

If the queue you are searching for is not displayed in the list, answer these questions for the upstream (primary) cluster:

  • Is the virtual host tagged with standby_replication?
  • Does the effective policy for the queue have the definition remote-dc-replicate: true?
  • Is the queue type Quorum or Classic?
  • Do the internal streams for the virtual host have any messages? These are rabbitmq.internal.osr.messages and rabbitmq.internal.osr.metrics on each replicated virtual host.

  • Can the replicator user for WSR authenticate? Verify this by running:

    rabbitmqctl authenticate_user some-user some-password

If the answers for all these questions are what they should be, search the downstream (standby) cluster RabbitMQ logs for any related errors.

For streams, messages are continually transferred to the downstream (secondary) cluster into the destination streams. These messages are visible in the RabbitMQ Management UI automatically (before promotion). You can also see them by running:

rabbitmqctl list_queues

If the messages you are searching for are not displayed in the streams, answer these questions for the upstream (primary) cluster:

  • Is the virtual host tagged with standby_replication?
  • Does the effective policy for the stream have the definition remote-dc-replicate: true?
  • Is the queue type Stream?

  • Can the replicator user for WSR authenticate? Verify this by running:

    rabbitmqctl authenticate_user some-user some-password

Verify that the downstream (standby) cluster has received the replicated messages before promotion

Before running the promotion command (mentioned in the earlier Promote the downstream (standby) cluster for disaster recovery section), you can verify which queues have messages and acknowledgements recovered. The exact number of messages that are collected can also be verified.

Accessing the replication information can be resource-intensive. The following operations can take a long time to finish when the amount of data to recover is substantial.

To verify which classic queues and quorum queues have messages available for recovery, run the following command in the downstream (standby) cluster:

rabbitmq-diagnostics inspect_standby_downstream_metrics
# Inspecting standby downstream metrics related to recovery...
# queue  timestamp  vhost
# ha.qq  1668785252768  /

For quorum queues and classic queues, you can inspect the number of messages, their routing key, and the virtual host from the RabbitMQ Management UI. For more information, see Method 1 in the Verify Warm Standby Replication is configured correctly section.

Alternatively, you can access the same information by running the following command in the downstream (standby) cluster:

rabbitmq-diagnostics inspect_local_data_available_for_standby_replication_recovery
# Inspecting local data replicated for multi-DC recovery
# exchange  messages  routing_key	vhost
# myexchange  2	demo  /
#   2	ha.qq	/

The previous operation (using either method) reports how many messages can be published to a specific exchange, with a specific routing key. If the exchange is empty or missing, it means that the message was published to the default exchange. The routing key might not be the same as the name of the queue.

The same applies to streams. You can inspect the number of messages, the stream, and the virtual host from the RabbitMQ Management UI. For more information, see Method 1 in the Verify Warm Standby Replication is configured correctly section for details.

Alternatively, run the following command in the downstream (standby) cluster:

rabbitmq-diagnostics inspect_local_stream_data_available_for_standby_replication_recovery
# Inspecting local stream data replicated for multi-DC recovery
# messages  name      vhost
#        7  stream.1  /

How to get a license

To get a license for Tanzu RabbitMQ products, fill in the Tanzu RabbitMQ support contact form and we will get back to you with a tailored quote.