AWS Compute Blog

Implementing Federation on Amazon MQ for RabbitMQ Private Brokers

Federation in RabbitMQ helps in message exchange and flow across multiple RabbitMQ brokers. Amazon MQ for RabbitMQ allows federated exchanges and queues via the Federation Plugin. The federation plugin enables a downstream broker to consume a message from an exchange or a queue on an upstream. This is used to connect multiple RabbitMQ brokers and provides multiple benefits like scalability, allowing to scale out the messaging infrastructure horizontally across multiple nodes or clusters. It also provides high availability for message replication across brokers for redundancy and the ability to segregate based on security or other criteria. These benefits allow federation to be used for the below use cases:

  1. Multi region deployments
  2. Hybrid cloud deployments
  3. Disaster recovery
  4. Migrating from on-premises to cloud

Currently, the federation plugin on Amazon MQ for RabbitMQ connects to publicly available upstream brokers only. This post explains how to implement federation for Amazon MQ RabbitMQ Private Brokers using Network Load Balancers (NLB). The steps allow private brokers to communicate with each other to create a distributed system.

Overview

In this solution, you will use two single-instance brokers to implement federation with private brokers.

  1. Create two Amazon Virtual Private Cloud (VPC) – one for upstream broker and one for downstream broker. Each VPC has a private and a public subnet along with internet gateway, security groups, route tables.
  2. Create Amazon MQ RabbitMQ private brokers in the private subnets of each VPC. The broker actually resides in an account that is owned by the Amazon MQ Service, in a private subnet with a Network Load Balancer (NLB) in front of it. The NLB is used to access the broker from your account using the Elastic Network Interface (ENI) associated with the VPC Endpoint for the NLB.
  3. Create a NLB pointing to the ENI for the upstream broker. The security group associated with the NLB is used to restrict traffic to only the NAT IPs associated with the downstream broker. The upstream broker that was accessible only privately will now be connected to the public internet with IP allow listing and messages will potentially transit the internet.
  4. Create an Amazon EC2 Instance in the downstream VPC in the public subnet to connect to it and setup the federation. You need the EC2 instance only for the setup and testing.
  5. Send a message to the upstream broker using the NLB endpoint, the message is also available to the downstream broker for consumption.

Prerequisites

The following are the prerequisites for this setup:

  • Access to an AWS account.
  • An AWS IAM user/Principal with the required permissions to deploy the infrastructure.

The stack creates two new VPC. Make sure that you have fewer than five VPCs in the selected region. You increase this limit using Quotas.

Deploying the solution

You will deploy the solution using AWS CloudFormation:

The high-level steps are the following:

  1. Deploy the broker CFN stack to create VPCs, subnets, internet gateway, security groups and route tables, along with the Amazon MQ RabbitMQ brokers
  2. Get the IP address of the private upstream broker created in the broker stack
  3. Open AWS support case to get the IP to allow for the NLB
  4. Create an NLB Stack with the Network Load Balancer and rules for accessing it using AWS CloudFormation
  5. Set up the federation between the Amazon MQ RabbitMQ brokers and testing the setup

This solution is available on GitHub in the AWS Samples repository.

Step 1: Deploy the AWS CloudFormation template for the broker stack

  1. Go to the CloudFormation Console and choose Create Stack. Choose With new resources (Standard) from the drop down.
  2. For Prepare template, choose an existing template and then for Specify template, choose Upload a template file and use this template file
  3. Provide a Stack name (such as BrokerStack).
  4. Update the username and CIDR Blocks provided as parameters to the stack or leave them as defaults. For ease of setup, this template uses EC2 with managed prefix lists for EC2 Instance Connect for five regions: us-east-1, us-west-1, us-west-2, eu-west-1 and ap-south-1. Add prefix lists for other regions in the template to run this cloud formation template in those regions.
  5. Choose Next and leave everything else as defaults.
  6. Choose Submit.

The broker stack deployment takes 10 -15 minutes.

The template creates two VPCs along with a private and public subnet on each VPC with internet gateway, security groups and route tables. It also creates two private brokers in each VPC along with an EC2 Instance (t2.micro) on the downstream VPC.

Step 2: Retrieve the IP Address for the private upstream broker

  1. Once the above stack creation is complete, navigate to the Outputs tab for the stack and copy the output for PrivateUpstreamBrokerEndpoints.
  2. Extract only the host name from the “PrivateUpstreamBrokerEndpoints” in the output from above.
  3. Resolve the hostname using the following commands.
    Linux or Mac

    $ dig +short {hostname}

    Windows

    C:\> nslookup {hostname}

Take note of the IP address. You will use it in later steps.

Step 3: Create a support case to get the Amazon MQ Rabbit MQ Downstream Broker NAT IPs

Create a support case with AWS Support to get the NAT IPs associated with the downstream MQ Broker. Provide the broker Amazon Resource Name (ARN) and explain your use case and the need to do federation allow listing in the description. Use this IP address to allow the Network Load Balancer to be accessed from particular IPs only.

Step 4: Deploy the AWS CloudFormation template for NLB Stack

  1. Go to the CloudFormation Console and choose Create Stack. Choose With new resources (Standard) from the drop down.
  2. For Prepare template, Choose an existing template. For Template source, choose Upload a template file and choose this template file.
  3. Choose Next.
  4. Under Specify stack details provide a Stack name (such as NLBStack).
  5. Use the IP Address from Step 2 and Step 3 above in the parameters and choose Next.
    Make sure that the NAT IP Address is a valid CIDR range like 52.0.0.1/32.
  6. Keep the rest as defaults and choose Next again
  7. Choose Submit.

The template creates a Network Load Balancer with 2 target groups and a Security Group for it and adds rules to the Upstream Default Security group.

Step 5: Configure Federation in the downstream broker

  1. Use the Upstream Broker NLB URL output from the NLBStack and replace it in the following export commands along with the Downstream Broker Uri from the output of the BrokerStack. 
    export Upstream_Broker_NLB= <UpstreamBrokerNLBURL>
export Downstream_Broker_Uri= <DownstreamBrokerURI>
  2. From the AWS Console, search for AWS Secrets Manager and choose Secrets. You will find 2 secrets with names as DownstreamBrokerUsernamePassword and UpstreamBrokerUsernamePassword. Open one of them and choose Retrieve Secret value to get the passwords and usernames for the brokers. Repeat for the other one.
  3. Replace values for Upstream_Broker_Username, Upstream_Broker_Password, Downstream_Broker_Username and Downstream_Broker_Password in the following commands. 
    ##creates federation on the private downstream broker
curl -XPUT -d'{"value":{"uri":"amqps://Upstream_Broker_Username:Upstream_Broker_Password@'"$Upstream_Broker_NLB"':5671","expires":3600000}}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/parameters/federation-upstream/%2f/my-upstream

##creates policy for federation on the private downstream broker with pattern for exchange with Test in its name
curl -XPUT -d'{"pattern":"^Test", "definition":{"federation-upstream-set":"all"},"apply-to":"exchanges"}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/policies/%2f/federate-me
  4. From the EC2 Console, select the EC2 instance created as part of the Broker Stack in Step 1. Choose Connect and login to the instance using EC2 Instance Connect. Once connected to the terminal, paste the above lines with replaced values to create the federation upstream and the policy associated with it.

Step 6: Create TestExchange and Test Queue and Bind them

  1. Run the following steps to create a test exchange, a queue, and the binding for them. Replace values for Downstream_Broker_Username and Downstream_Broker_Password. 
    ##creates a test exchange on the private downstream broker
curl -H "content-type:application/json" -XPUT -d'{"type":"fanout","durable":true}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/exchanges/%2f/TestExchange

##creates a test queue on the private downstream broker
curl -H "content-type:application/json" -XPUT -d'{"durable":true,"arguments":{"x-dead-letter-exchange":"", "x-dead-letter-routing-key": "my.queue.dead-letter"}}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/queues/%2f/TestQueue

##Binds the queue to the exchange on the private downstream broker
curl -H "content-type:application/json" -XPOST -d'{"routing_key":"","arguments":{}}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/bindings/%2f/e/TestExchange/q/TestQueue

Step 7: Validate Federation Status and Test Federation between brokers

  1. Check the Federation status by running the following command while still connected to the EC2 in the same session. Replace values for Downstream_Broker_Username and Downstream_Broker_Password. 
    ##check federation status on the private downstream broker and format it as JSON
curl -XGET https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/federation-links | python3 -m json.tool

    The output will look like the below with status as running.

    [
    {
        "node": "rabbit@localhost",
        "exchange": "TestExchange",
        "upstream_exchange": "TestExchange",
        "type": "exchange",
        "vhost": "/",
        "upstream": "my-upstream",
        "id": "5cd2293f",
        "status": "running",
        "local_connection": "<rabbit@localhost.1746117897.30989.0>",
        "uri": "amqps://MyUpstreamNLB-XXXXXXXX.elb.us-east-1.amazonaws.com:5671",
…
    }
]
  2. (Optional) Send a test message now. Since you restricted the Upstream Broker NLB to only receive traffic from the Downstream broker (via the IP Address received from the support case), you will need to manually allow the EC2 Public IP Address in the NLB Security Group that was created for port 443 to perform the below step. You will also need to allow the egress from EC2 to access the NLB. 
    ##Send test message on the upstream broker
curl -k -H "content-type:application/json" -XPOST -d'{"properties":{},"routing_key":"MYKEY","payload":"Hello World","payload_encoding":"string"}' https://Upstream_Broker_Username:Upstream_Broker_Password@{$Upstream_Broker_NLB}/api/exchanges/%2f/TestExchange/publish

    Once the message is sent it will show up as routed: true. This means that the message routed to the downstream broker successfully.

  3. Use the following command to validate the message on the downstream broker. This should show the payload that you sent earlier. 
    ## Get message from queue on the downstream broker
curl -H "content-type:application/json" -XPOST -d'{"ackmode":"ack_requeue_true","count":1,"encoding": "auto"}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/queues/%2f/TestQueue/get

    Output:

    [
    {
        "payload_bytes": 11,
        "redelivered": true,
        "exchange": "TestExchange",
        "routing_key": "MYKEY",
        "message_count": 0,
         …
        "payload": "Hello World",
        "payload_encoding": "string"
    }
]

Cleanup

This section provides information for deleting various resources created as part of this post.

  1. Delete Stack NLBStack created as part of Step 4. For instructions, refer to Deleting a stack on the AWS CloudFormation console.
  2. Delete the BrokerStack created in Step 1.

Conclusion

This post explained how to implement federation for Amazon MQ RabbitMQ private brokers. You can extend this solution to RabbitMQ brokers in a cluster deployment, same as a single-instance broker. With federated exchanges, you can create a distributed system of RabbitMQ brokers to improve reliability and scalability of the messaging system. You can also use this as a template for hybrid architecture to move messages from a private on-premises broker to the cloud as explained in Migrating message driven applications to Amazon MQ for RabbitMQ. Get more details on Federation plugin from official documentation of RabbitMQ. Get more details on Amazon MQ for RabbitMQ in our developer guide.