Skip to main content

AWS Solutions Library

Overview

This Guidance demonstrates how to securely transfer Amazon Simple Storage Service (Amazon S3) objects and Amazon Elastic Container Registry (Amazon ECR) images across AWS environments. Through an intuitive user interface, you can easily create and manage various types of transfer tasks between AWS partitions (aws, aws-cn, aws-us-gov) and from other cloud providers to AWS. The Guidance provides scalable and trackable data transfers, streamlining the process of moving both Amazon S3 objects and Amazon ECR images between environments.

Please note: If you deploy this solution in the AWS (Beijing) Region operated by Beijing Sinnet Technology Co., Ltd. (Sinnet), or the AWS (Ningxia) Region operated by Ningxia Western Cloud Data Technology Co., Ltd., you are required to provide a domain with ICP Recordal before you can access the web console.

The web console is a centralized place to create and manage all data transfer jobs. Each data type (for example, Amazon S3 or Amazon ECR) is a plugin for Data Transfer Hub, and is packaged as an AWS CloudFormation template hosted in an Amazon S3 bucket that AWS owns. When you create a transfer task, an AWS Lambda function initiates the AWS CloudFormation template, and state of each task is stored and displayed in the DynamoDB tables.

As of today, the solution supports two data transfer plugins: an Amazon S3 plugin and an Amazon ECR plugin.

How it works

This architecture diagram illustrates how to secure, scale, and track data transfer for Amazon S3 objects and Amazon ECR images.

Overview

This architecture diagram illustrates how to secure, scale, and track data transfer for Amazon S3 objects and Amazon ECR images.

Download the architecture diagram PDF
Diagram of an AWS cloud architecture showing a data transfer workflow involving services like Amazon Cognito, AWS AppSync, AWS Lambda, Amazon S3, AWS Fargate, and Amazon DynamoDB, with interactions between a customer's AWS account and an AWS managed account.

ECR transfer option

This architecture diagram illustrates how to run the Amazon ECR plugin to transfer container images from other container registries.

Download the architecture diagram
Diagram of an AWS workflow involving EventBridge Scheduler, Lambda, Fargate, DynamoDB, SNS, Systems Manager Parameter Store, Secrets Manager, and ECR for container management and automation.

S3 transfer option

This architecture diagram illustrates how run the Amazon S3 plugin to transfer objects from their sources into S3 buckets.

Download the architecture diagram
A flow diagram illustrating an AWS architecture with services like AWS Lambda, Amazon EC2, Amazon S3, Amazon SQS, Amazon DynamoDB, and AWS Step Functions, showing data processing and scaling workflows.

Deploy with confidence

Everything you need to launch this Guidance in your account is right here

We'll walk you through it

Dive deep into the implementation guide for additional customization options and service configurations to tailor to your specific needs.

Open guide

Let's make it happen

Ready to deploy? Review the sample code on GitHub for detailed deployment instructions to deploy as-is or customize to fit your needs. 

Go to sample code


Use this sample code to deploy the Guidance using the command line interface (CLI)

Go to sample code

Well-Architected Pillars

The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.

Step Functions provides a reliable, scalable, and fault-tolerant workflow management system, helping ensure data transfer tasks are executed reliably. Amazon ECS and Amazon ECR simplify the deployment and management of containerized components, improving operational efficiency. These services enable DevOps best practices and promote a culture of continuous improvement and automation while reducing the operational overhead and manual effort required to manage infrastructure and data transfer workflows.

Read the Operational Excellence whitepaper

Amazon Cognito and OpenID Connect providers help ensure that only authorized users can access and manage data transfer tasks. AWS AppSync provides a secure GraphQL interface to interact with the backend APIs, protecting against unauthorized access. DynamoDB offers secure storage for data transfer task details, with options for encryption at rest. These services provide a comprehensive, end-to-end secure workflow for managing data transfers across AWS partitions and external cloud providers, protecting sensitive data from unauthorized access or tampering.

Read the Security whitepaper

Amazon ECS and Amazon ECR offer a highly available and scalable way to run and manage container-based components. Amazon Simple Notification Service (Amazon SNS) provides a reliable notification mechanism. These services ensure you can reliably and consistently provision and manage the necessary infrastructure for data transfers, leveraging the scalability and high availability of containerized services like Amazon ECS and Amazon ECR.

Read the Reliability whitepaper

Lambda provides a serverless, scalable, and highly performant compute service to run backend components. DynamoDB offers a fast and scalable NoSQL database to store data transfer task information, with low latency access. CloudFront improves performance and responsiveness of the web interface by caching and serving static assets from a global network of edge locations. These services leverage inherent performance and scalability benefits of serverless and managed AWS services, reducing operational overhead while ensuring the Guidance can handle increasing workloads without compromising performance.

Read the Performance Efficiency whitepaper

Lambda and the serverless architecture allow for on-demand, pay-as-you-go compute resources, eliminating the need for always-on server infrastructure. DynamoDB provides a pay-per-request NoSQL database service, meaning you only pay for the resources consumed, without incurring costs of provisioning and maintaining a database. Amazon SNS and Amazon SQS provide cost-effective serverless notifications, while Amazon ECS on Fargate enables on-demand infrastructure for containers. This serverless approach helps reduce operational costs and overhead associated with managing and scaling infrastructure.

Read the Cost Optimization whitepaper

The serverless design using Lambda, Amazon SQS, and DynamoDB aims at reducing carbon footprint compared to continually operating on-premises servers. Step Functions, Amazon SNS, and Amazon SQS enable serverless notifications, while Amazon ECS on Fargate provides on-demand infrastructure for containers. These serverless and cloud-native services enable you to scale your infrastructure efficiently, reduce required resources and related carbon footprints as compared to on-premises servers, and optimize resource utilization.

Read the Sustainability whitepaper

Disclaimer

The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.