Containers

Streamline service-to-service communication during deployments with Amazon ECS Service Connect

When deploying containerized microservices, maintaining reliable service discovery and efficient routing during updates presents significant challenges. Traditional blue/green deployment approaches rely heavily on load balancer for traffic management, which can become complex when dealing with container-based service-to-service communication. This complexity increases the possibility of service disruption and makes it difficult to test new versions in isolation before directing production traffic to them.

Amazon Elastic Container Service (Amazon ECS) combined with blue/green deployments using Amazon ECS Service Connect offers a direct solution to these challenges. This integration enhances the traditional blue/green deployment model by introducing test traffic routing capabilities directly within a shared namespace. You can now deploy new versions of your microservices with near-zero downtime, test them thoroughly in isolation, and maintain the ability to roll back quickly if needed. In this post, we demonstrate how to implement this powerful combination to create a more streamlined and resilient deployment process for your containerized applications.

Feature overview

Blue/green deployments maintain two identical, but separate, environments: blue for the current production version and green for the new version. Unlike rolling updates, both versions run simultaneously with controlled traffic shifting between them, enabling quick rollbacks and near-zero downtime transitions.

ECS Service Connect streamlines containerized application communication through managed service discovery and automatic sidecar proxy injection. In this way the services can easily discover and communicate with each other using direct, logical names within the shared namespace.

The combination of ECS Service Connect with blue/green deployments enhances traffic management through built-in service discovery and reliable routing. This allows teams to deploy with improved reliability and minimal service interruption.

Amazon ECS Service states for the blue/green deployment workflow

Figure 1: Amazon ECS Service states for the blue/green deployment workflow

The blue/green deployment process orchestrates through three distinct coordinated phases, as shown in the preceding figure, each designed to help safe and controlled transitions:

  1. Initial Deployment State: Both environments are deployed but traffic is not yet routed to the green tasks.
  2. In-progress Deployment State: Test traffic is shifted from the blue tasks to the green tasks. When validation succeeds, production traffic is also shifted to the green tasks though automated configuration changes in the ECS Service Connect proxies.
  3. Final Deployment State: During the configurable bake time, both environments remain scaled to enable quick rollback if needed.

The deployment workflow is governed by lifecycle stages, which are precise points in the deployment process that enable fine-grained control and validation. Each stage represents a specific deployment event (for example POST_TEST_TRAFFIC_SHIFT, PRE_PRODUCTION_TRAFFIC_SHIFT) and can trigger associated lifecycle hooks. These hooks are AWS Lambda functions that execute validation checks or a custom logic.

For example, a POST_TEST_TRAFFIC_SHIFT hook could validate the green environment by simulating test traffic before shifting the entire production traffic. This validation process provides higher deployment reliability by enforcing quality gates at each phase while minimizing the risk of service disruption. For further information about Amazon ECS blue/green lifecycle stages, refer to the AWS Documentation.

Understanding blue/green deployments with ECS Service Connect

ECS Service Connect streamlines blue/green deployments by managing version transitions through embedded routing capabilities within Amazon ECS Tasks. Unlike traditional approaches that need DNS updates and load balancer reconfigurations, ECS Service Connect handles routing through a sidecar proxy, which creates a clear transition.

During blue/green deployments, ECS Service Connect employs a header-based routing mechanism that enables precise traffic control between versions. Your application code remains unchanged while the infrastructure handles the version targeting. The following is an example of how to configure the testTrafficRules with a specific header name ($HEADER) and header value ($VALUE).

"serviceConnectConfiguration": {
    /* ... */
    "testTrafficRules": {         // Routes test traffic to green deployment
        "header": {               // Header-based routing for test traffic
            "name": "$HEADER",    // Custom Header Name
            "value": {
                "exact": "$VALUE" // Exact match for header value
            }
        }
    }
}

The following is an example configuration for a POST_TEST_TRAFFIC_SHIFT lifecycle hook. The complete list of available lifecycle stages can be found in the AWS Documentation.

"deploymentConfiguration": {
    "strategy": "BLUE_GREEN",
    "bakeTimeInMinutes": 2,       // Configurable bake time for blue environment
    "lifecycleHooks": [.          // Lifecycle hooks configuration
        {
            "hookTargetArn": "$AWS_LAMBDA_FUNCTION",   // AWS Lambda function ARN
            "roleArn": "$ECS_LAMBDA_INVOKE_ROLE",  // Role to invoke the AWS Lambda
            "lifecycleStages": [
                "POST_TEST_TRAFFIC_SHIFT"          // Lifecycle stage
            ]
        }
    ]
}

ECS Service Connect changes how deployments work by introducing an embedded proxy layer that helps separate your application code from deployment. Developers can focus on building features without deployment concerns, while operations teams can implement improved release strategies without touching application code. This separation eliminates the back-and-forth coordination typically needed between teams and streamlines the entire release process for containerized applications.

Blue/green test strategies

A critical phase in blue/green deployments is validating the green environment before switching the production traffic. Within ECS Service Connect, test traffic is routed exclusively within the namespace, which necessitates careful consideration when implementing testing patterns. In this section we explore two common patterns:

Header-based testing through Application Load Balancer: This approach uses the Application Load Balancer (ALB’s) header pass through capabilities. The frontend service forwards specific headers to backend services, enabling targeted testing of the green environment through the load balancer while still isolating production traffic. This method provides a direct mechanism for controlling traffic flow and validating service behavior. To avoid exposing the testing path to end users, we recommend enhancing this solution by using a dedicated load balancer not exposed to the internet and deploying the Lambda functions within the application VPC. In the walkthrough section we implement the basic header-based routing approach.

Header-based testing through ALB workflow

Figure 2: Header-based testing through ALB workflow

Dedicated test service implementation: This method involves deploying a dedicated testing service within the namespace. In the following diagram, a POST_TEST_TRAFFIC_HOOK scales up an Amazon ECS service to validate the green deployment by using the test header within the namespace. The results of the test are passed to an Amazon S3 bucket. Logic in the function would evaluate this outcome. This approach provides comprehensive validation capabilities while maintaining the integrity of the architecture. However, it does need more AWS resources.

Dedicated test service implementation workflow

Figure 3: Dedicated test service implementation workflow

These validation strategies enable thorough testing of new versions while preserving the security and isolation benefits of ECS Service Connect, making sure of a reliable deployment processes.

Prerequisites

To deploy this walkthrough, you need access to an AWS account with the necessary permissions to create the resources, the AWS Command Line Interface (AWS CLI), and the AWS Cloud Development Kit (AWS CDK).

Throughout this walkthrough we deploy resources to an ECS cluster and into an Amazon Virtual Private Cloud (Amazon VPC). The code for these step-by-step examples can be found in an AWS CDK app within our sample GitHub repository.

Walkthrough

Having explored the core concepts of ECS Service Connect and its blue/green deployment strategies, we can demonstrate these principles in action through a practical implementation.

This guide demonstrates how to deploy a sample application using blue/green deployment with ECS Service Connect and using AWS CLI. The configuration can also be implemented through AWS CloudFormation, the AWS Management Console, or other infrastructure as code (IaC) tools.

1. Set up the environment

The environment setup process involves deploying the necessary AWS infrastructure and configuring the initial services. Follow these sequential steps to establish the necessary resources:

  1. Clone the GitHub repository.
$ git clone https://github.com/aws-samples/sample-amazon-ecs-blue-green-deployment-patterns.git
  1. Build the Lambda function to be configured as lifecycle hook.
$ cd sample-amazon-ecs-blue-green-deployment-patterns/ecs-bluegreen-service-connect
$ sh build-lambda-zip.sh
  1. Deploy the core infrastructure using AWS CDK, which creates a CloudFormation stack containing the Amazon VPC, ALB, ECS cluster, Lambda function, and associated AWS Identity and Access Management (IAM) roles.
$ npm install
$ cdk bootstrap
$ cdk deploy
  1. After infrastructure deployment, configure the environment variables from the CloudFormation stack outputs
$ source load-env-variables.sh
  1. Deploy the initial version of the ECS Services and create the necessary task definitions:
$ sh demo-setup.sh

The demo-setup.sh script generates an outputs directory containing JSON configuration files for task definitions and service configurations, which are used in subsequent deployment steps. These files serve as templates for the blue/green deployment configuration used in the following sections.

2. Review the environment

This deployment creates an architecture that supports blue/green deployments using ECS Service Connect with the following components:

  • ALB as the primary entry point, managing and distributing external traffic to the bluegreen-frontend service.
  • ECS cluster with two interconnected Amazon ECS Services:
    • bluegreen-frontend contains an nginx-based reverse proxy that processes incoming requests from the ALB and forwards them on with the headers intact. In production this frontend service would contain application logic, but for simplicity in this walkthrough we just forward traffic.
    • bluegreen-backend processes requests and serves static HTML content.
  • ECS Service Connect for service discovery and routing between components.
  • Lambda function implements deployment validation through lifecycle hooks. This function activates during the POST_TEST_TRAFFIC_SHIFT stage, performing critical validation when test traffic routes to the green environment, helping deployment reliability.

The following architecture diagram shows the initial state before deploying the green version of the bluegreen-backend service.

Initial state before deploying green version of the application

Figure 4: Initial state before deploying green version of the application

The deployment process uses a Lambda function to validate the POST_TEST_TRAFFIC_SHIFT lifecycle stage, implementing the verification of the green environment. This function executes a validation by sending HTTP requests with the specified test header (x-amzn-ecs-bluegreen-test:test) to the ALB endpoint ($ALB_URL).

Based on the validation results, the Lambda function returns one of three possible hookStatus values, each triggering specific deployment behavior:

  • SUCCEEDED: Allows deployment to proceed
  • FAILED: Triggers rollback
  • IN_PROGRESS: Requests more validation attempts, implementing a retry mechanism with a delay between attempts (default 30 seconds)

The following implementation code demonstrates the Lambda function’s validation logic:

import requests
def lambda_handler(event, context):
    """
    Tests blue/green deployment.
    """
    try:
        response = requests.get(
            $ALB_URL, 
            headers={"x-amzn-ecs-bluegreen-test": "test"},
            timeout=5
        ) 
        if response.status_code == 200 and "Green Version" in response.text:
            return {"hookStatus": "SUCCEEDED"}
        else:
            return {"hookStatus": "FAILED"}
    except Exception as e:
        print(f"Error: {str(e)}")
        return {"hookStatus": "FAILED"}

Having established the baseline architecture and validation mechanisms, we proceed with updating the bluegreen-backend service to implement the green version.

2. Update the backend task to the green version

With our environment configured, we can create an updated version of the bluegreen-backend service to serve as the green deployment. The modification involves registering a new task definition that changes the background color from blue to green. The new background color provides a clear visual indicator of successful traffic routing during the deployment process.

Execute the following command to register the new task definition, using the JSON file that was previously generated during the environment setup phase:

$ aws ecs register-task-definition \
    --cli-input-json file://outputs/taskdef_backend_update.json

3. Deploy the new version of the application using the blue/green strategy

With the new task definition registered, we can implement the blue/green deployment using ECS Service Connect. This section covers the necessary deployment configuration parameters, such as test traffic rules and lifecycle hooks that enable controlled testing and validation.

The deployment necessitates updating the bluegreen-backend service configuration to incorporate both the new task definition and the necessary traffic routing rules: serviceConnectConfiguration and deploymentConfiguration.

The serviceConnectConfiguration section: It needs testTrafficRules to define precise traffic routing during the deployment process. These rules specify which requests should be directed to the green environment based on HTTP headers, allowing isolated testing of the new version.

"serviceConnectConfiguration": {
    ...
    "services": [{
        ...
        "clientAliases": [{
            ...
            "testTrafficRules": {
                "header": {
                    "name": "x-amzn-ecs-bluegreen-test",
                    "value": {
                        "exact": "test"
                    }
                }
            }
        }]
    }]
},

The deploymentConfiguration section: It needs specific parameters to implement the blue/green deployment strategy with appropriate validation controls. The configuration defines how Amazon ECS manages the deployment process and validates each stage through the Lambda function integration. The hookTargetArn specifies the validation of the Lambda function, roleArn provides the necessary IAM permissions, and POST_TEST_TRAFFIC_SHIFT defines when the deployment validation checks are executed.

"deploymentConfiguration": {
    "strategy": "BLUE_GREEN",
    "bakeTimeInMinutes": 2,
    "lifecycleHooks": [{
        "hookTargetArn": "arn:aws:lambda:us-west-2:1111111111:function:LifeCycleHook",
       "roleArn": "arn:aws:iam::1111111111:role/EcsLambdaInvokeRole",
        "lifecycleStages": [
            "POST_TEST_TRAFFIC_SHIFT"
        ]}
    ]
}

Run the following command to deploy the green version of the bluegreen-backend service.

$ aws ecs update-service \
    --service bluegreen-backend \
    --cli-input-json file://outputs/service_backend_update.json

4. Understand the deployment process

During a blue/green deployment with ECS Service Connect, Amazon ECS initiates the process by creating new green environment tasks, and when it is ready the test traffic is directed to them through the POST_TEST_TRAFFIC_SHIFT lifecycle hook. The validation process involves Lambda sending requests with test headers to the ALB, which forwards them to the frontend service, while ECS Service Connect handles routing to the green backend. The deployment’s success depends on the Lambda function’s validation results, which either allows the deployment to proceed (SUCCEEDED) or triggers a rollback (FAILED).

ECS Service Connect Blue/Green deployment steps

Figure 5: ECS Service Connect Blue/Green deployment steps

This validation mechanism helps to validate that the green environment is operating correctly before any production traffic transition occurs. For detailed specifications of the deployment process and more configuration options, refer to the AWS documentation.

5. Verify the new version of the application

After the blue/green deployment completes, verify that the new green version is successfully serving production traffic by executing the following command:

$ curl -s $ALB_DNS | grep "Green Version"

This command sends a request to the ALB endpoint (without test header) and searches for the “Green Version” text in the response. A successful response indicates that production traffic is now being served by the green environment, which confirms that the deployment has completed successfully and the traffic transition has occurred as expected.

Cleaning up

After completing this walkthrough, you must clean up all deployed resources to prevent ongoing charges and maintain a tidy AWS environment. This step prevents unexpected charges and keeps your AWS environment tidy by eliminating unused resources.

  1. Remove all resources created during the demonstration setup:
$ sh demo-teardown.sh
  1. Destroy the AWS CDK-deployed infrastructure components:
$ cdk destroy
  1. Retrieve the task definition Amazon Resource Names (ARNs):
$ aws ecs list-task-definitions --family-prefix bluegreen-backend \
    --query 'taskDefinitionArns[*]'

$ aws ecs list-task-definitions --family-prefix bluegreen-frontend \
    --query 'taskDefinitionArns[*]'
  1. Remove the Amazon ECS Task Definitions extracted by the previous command by replacing the $TASK_DEFINITION_ARN variable.
$ aws ecs deregister-task-definition \
    --task-definition $TASK_DEFINITION_ARN 

$ aws ecs delete-task-definitions \
    --task-definitions $TASK_DEFINITION_ARN 

Conclusion

Amazon ECS Service Connect with blue/green deployments modernizes containerized microservice deployments by streamlining service discovery, enabling isolated testing, and providing reliable traffic shifting with rollback capabilities. We’re excited to see how this feature can improve your deployment processes and reduce operational risks. Check out the AWS Documentation to implement this feature with your Amazon ECS Services and share your feedback as you integrate this capability into your applications.

About the authors

Henrique Santana is a containers specialist with nearly 20 years in infrastructure operations who helps organizations modernize their technology stacks through container adoption and orchestration solutions. He has guided numerous enterprises in overcoming containerization challenges, resulting in improvements in operational efficiency and accelerated time-to-market. When not optimizing container environments, he shares insights from the frontlines of infrastructure to help businesses navigate their cloud-native journeys.

Simone Tallevi-Diotallevi is a Partner Solution Architect who empowers EMEA consulting firms to architect and implement cutting-edge AWS cloud solutions. Through strategic technical guidance, he enables both partners and customers to maximize their cloud investments and adopt industry best practices. Leveraging over 15 years of solution development and design experience, he serves as a trusted advisor to enterprises navigating their cloud transformation initiatives. His collaborative approach combines deep technical knowledge with business acumen to help organizations modernize their infrastructure and achieve measurable business outcomes.