AWS Storage Blog

Cross-account Amazon S3 bulk transfers with enhanced AWS KMS support

Cross-account Amazon S3 bulk transfers with enhanced AWS Key Management Service (AWS KMS) support become increasingly critical as organizations grow and accumulate vast amounts of digital assets across their enterprise. Managing millions or even billions of files presents unique challenges, especially when these files need to be moved securely between different AWS accounts. Operations such as copying encrypted files between departments or external partners can become overwhelming when performed manually or with basic tools. The complexity increases exponentially with the volume of data, making what should be routine transfers into significant technical challenges.

Amazon S3 offers powerful capabilities for handling these large-scale file operations efficiently. Amazon S3 Batch Operations can perform actions across billions of objects (up to 20 billion) and petabytes of data using a single request. This is particularly valuable for enterprises with multiple accounts across departments, each maintaining their own storage resources. When sensitive data needs to move between these accounts—whether between internal departments or with external partners—S3 Batch Operations provides a streamlined approach that maintains proper security controls, such as support for encrypted data using AWS KMS Customer Managed Keys (CMKs).

In this post, I demonstrate how to build a repeatable process for cross-account bulk data transfers with comprehensive encryption support. I provide both a one-click AWS CloudFormation deployment option for quick setup and detailed manual instructions for those who need more control. This approach addresses critical security requirements that enterprises face when transferring sensitive data across organizational boundaries, providing proper permissions and encryption handling throughout the process.

Solution overview

S3 Batch Operations is a feature designed specifically to perform operations on large numbers of S3 objects with a single request. Unlike custom scripts or applications that necessitate you handle object listing, error handling, and operation tracking, S3 Batch Operations manages the entire workflow automatically. It accepts a list of objects (either from an S3 Inventory report or a custom CSV manifest), performs the specified operation on each object, tracks progress, and generates comprehensive completion reports.

For cross-account transfers, S3 Batch Operations eliminates the need to write complex code that handles authentication, permissions, and throttling. It seamlessly integrates with AWS Identity and Access Management (IAM) for security controls and supports AWS KMS encryption for sensitive data. This makes it the ideal solution for enterprises that need to move large volumes of data between accounts while maintaining security, compliance, and operational efficiency.

This post covers two approaches for providing the object list to S3 Batch Operations:

Approach A: Using an S3 Inventory report delivered to the source account to copy objects across AWS accounts
Approach B: Using a manually created CSV manifest file stored in the source account to copy objects across AWS accounts

This post includes:

  • Enhanced AWS KMS support: Complete permissions for copying AWS KMS-encrypted objects across accounts
  • One-click deployment: CloudFormation template for automated setup with approval workflow
  • Comprehensive security: Detailed IAM policies for cross-account access with encryption
  • Manual setup instructions: Step-by-step guide for customers who prefer manual configuration
  • Flexible manifest options: Support for both S3 Inventory reports and custom CSV manifests

Prerequisites

To follow along with the process outlined in this post, you need the following prerequisites:

  • A source AWS account with at least one S3 bucket containing objects to transfer
  • A destination AWS account with at least one bucket to store transferred objects
  • A job completion report bucket to store S3 Batch Operations copy job results, if you want to store job results
  • For Approach A: An S3 bucket containing the S3 inventory report in CSV format listing the objects to be copied
  • For Approach B: An S3 bucket containing the manually created CSV manifest file with the list of objects to copy
  • If you are using AWS KMS encryption: Access to modify AWS KMS key policies in the source account
  • Administrative permissions in both accounts to create IAM roles and modify S3 bucket policies

Approach A: Using Amazon S3 Inventory report

S3 Inventory provides a scheduled alternative to the Amazon S3 synchronous List API operation. S3 Inventory generates inventories of the objects in a bucket and their corresponding metadata on a daily or weekly basis. The resulting list is published to an output file in a bucket that you specify.

Benefits of using S3 Inventory:

  • Cost-effective: S3 Inventory can generate a list of 100 million objects for only $0.25 in the N. Virginia AWS Region
  • Comprehensive: Includes object metadata such as size, last modified date, storage class, and encryption status
  • Automated: Runs on a schedule without manual intervention
  • Scalable: Can handle billions of objects efficiently

Generate S3 Inventory report for cross-account data copy

  1. Configure S3 Inventory in source account:
    • In the Amazon S3 console on the source AWS account, choose the bucket with objects that you want to copy.
    • On the Management tab, choose Inventory, Add New.
    • For Output format, choose CSV and complete the other optional fields.
    • Set Daily frequency for report deliveries to get the first report sooner.
    • When entering information for the destination bucket, choose This account.
    • Enter the name of the destination inventory report bucket and optionally the prefix.
    • Choose Save.
  2. Configure inventory report bucket policy:
    • After saving the inventory configuration, the console displays a message about bucket policy requirements.
    • Copy the displayed bucket policy for the destination bucket.
    • Add this policy to the destination inventory report bucket.
  3. Wait for inventory generation:
    • Amazon S3 can take up to 48 hours to deliver the first report.
    • For automated notifications when reports are complete, implement the solution from knowing when an inventory is complete

Example S3 Inventory report structure:

"my-source-bucket","object-key","version-id","is-latest","is-delete-marker","size","last-modified-date","e-tag","storage-class","is-multipart-uploaded","replication-status","encryption-status","object-lock-retain-until-date","object-lock-mode","object-lock-legal-hold-status"
"my-source-bucket","documents/file1.pdf","null","true","false","1024","2023-01-15T10:30:00.000Z","d41d8cd98f00b204e9800998ecf8427e","STANDARD","false","","SSE-S3","","",""
"my-source-bucket","images/photo1.jpg","null","true","false","2048","2023-01-16T14:22:00.000Z","098f6bcd4621d373cade4e832627b4f6","STANDARD","false","","SSE-KMS","","",""

Approach B: Using manually created CSV manifest file

A manifest file is an S3 object listing object keys that you want to process**.** This approach gives you precise control over which objects to transfer. It is ideal when you need to transfer a specific subset of objects or when S3 Inventory is not suitable for your use case.

Benefits of using CSV manifest:

  • Precise control: Transfer only specific objects that you choose
  • Immediate availability: No waiting for inventory reports to generate
  • Custom logic: Apply business logic to determine which objects to transfer
  • Smaller scale: Ideal for targeted transfers of specific objects

CSV manifest file format

Each row in the file must include the bucket name, object key, and, optionally, the object version. The following are examples of manifest CSV files:

Without version IDs:

my-source-bucket,documents/file1.pdf
my-source-bucket,documents/file2.docx
my-source-bucket,images/photo1.jpg
my-source-bucket,data/dataset.csv

With version IDs (for versioned buckets):

my-source-bucket,documents/file1.pdf,3/L4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo
my-source-bucket,images/photo1.jpg,Rb2hVXcukU4Fqe4XiVrDXcDjlHL9a3UF

Creating your CSV manifest file

You can create the CSV manifest file using several methods:

  1. AWS Command Line Interface (AWS CLI): Use the list-objects-v2 command to generate the list:
aws s3api list-objects-v2 \ 
--bucket "BUCKET-NAME" \ 
--prefix "PREFIX" \ -
--query 'Contents[].{Bucket:`"BUCKET-NAME"`,Key:Key}' \ --output text | sed 's/\t/,/g' > manifest.csv

Omit the prefix option if it’s for whole bucket

  1. Custom script: Write a script that applies your business logic to determine which objects to include.
  2. Manual creation: For small numbers of objects, manually create the CSV file.

Manifest file storage considerations:

  • Store the manifest file in an S3 bucket accessible to your S3 Batch Operations job.
  • For cross-account scenarios, you can store it in either the source or destination account.
  • Make sure that proper permissions are configured for the batch operations role to read the manifest file.
  • Consider using S3 versioning on the bucket containing your manifest files.

Deployment options

Consider the following two deployment options.

Option 1: One-click CloudFormation deployment

For customers who want a streamlined deployment experience, we have a CloudFormation template that automates the entire setup process with built-in approval workflows. This template supports both S3 Inventory reports and CSV manifest files.

Features of the CloudFormation template:

How to deploy:

  1. Sign in to your destination AWS account where you want to copy the objects
  2. Choose the Launch Stack button for your preferred AWS Region:

  1. Fill in the necessary parameters:
    • Source account details and bucket names
    • Inventory manifest key:-
      • For Approach A: Path to your manifest.json file (such as my-inventory/2023-01-15T00-00Z/manifest.json)
      • For Approach B: Path to your CSV manifest file (such manifests/my-transfer-list.csv)
    • Destination side data, report bucket names and their KMS key ARN (if using KMS-CMK)
  1. Follow the approval workflow:
    • Confirm Amazon SNS subscription when prompted
    • Wait for detailed setup instructions through email
    • Apply the provided policies in your source account
    • Run the approval command to continue deployment
  2. Monitor and confirm:
    • Review the created S3 Batch Operations job
    • Confirm the job to start the transfer process

Option 2: Manual setup

This option is suitable, if you prefer manual setup or updating source bucket and KMS key policies may take time beyond CloudFormation (Option 1) timeout period

Step 1: Create IAM role in destination account

Navigate to the IAM console. Create a role in the destination account with the name S3BatchOperationsRole-CrossAccount:

  1. Trust policy: Allow S3 Batch Operations to assume the role.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "batchoperations.s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
  1. Permissions policy: Attach the following policy to the Batch operations copy role.
    • Amazon S3 access:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSourceBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAcl",
        "s3:GetObjectTagging",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::SOURCE-BUCKET-NAME",
        "arn:aws:s3:::SOURCE-BUCKET-NAME/*"
      ]
    },
    {
      "Sid": "AllowInventoryBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::INVENTORY-BUCKET-NAME",
        "arn:aws:s3:::INVENTORY-BUCKET-NAME/*"
      ]
    },
    {
      "Sid": "AllowDestinationBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectVersionAcl",
        "s3:PutObjectAcl",
        "s3:PutObjectVersionTagging",
        "s3:PutObjectTagging",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAcl",
        "s3:GetObjectTagging",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::DESTINATION-BUCKET-NAME",
        "arn:aws:s3:::DESTINATION-BUCKET-NAME/*"
      ]
    },
    {
      "Sid": "AllowReportBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::REPORT-BUCKET-NAME",
        "arn:aws:s3:::REPORT-BUCKET-NAME/*"
      ]
    }
  ]
}
    • AWS KMS key access: Add the following permissions to the S3 Batch Operations copy role on the destination side. The assumption here is that all of the buckets (source data bucket, manifest bucket, destination data bucket, and job completion report bucket) are using SSE-KMS CMK server-side encryption. Remove the permissions block/resources where the bucket is not using SSE-KMS CMK server-side encryption.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSourceBucketKMSAccess",
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:kms:source-bucket-region:SOURCE-ACCOUNT-ID::key/key-id"
      ]
    },
    {
      "Sid": "AllowManifestBucketKMSAccess",
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:kms:maifest-bucket-region:SOURCE-ACCOUNT-ID:key/key-id"
      ]
    },
    {
      "Sid": "AllowDestinationSideKMSAccess",
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": [
        "arn:aws:kms:destination-bucket-region:DESTINATION-ACCOUNT-ID:key/key-id",
        "arn:aws:kms:jobcompletionreport-bucket-region:DESTINATION-ACCOUNT-ID:key/key-id"
      ]
    }
  ]
}

Step 2: Configure source account permissions

2.1. Update the source bucket policy

  1. Sign in to the source account AWS Management Console and open the Amazon S3 console.
  2. Choose source bucket.
  3. Go to the Permissions tab.
  4. In Bucket policy, choose Edit.
  5. Add the following policy to your source data bucket by replacing destination account ID and source bucket name:

CONSOLE-USER-CREATING-JOB: User who creates batch job in destination account. If this user assumes a role in destination while creating the batch job, add that role ARN instead.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountBatchOperations",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::DESTINATION-ACCOUNT-ID:role/S3BatchOperationsRole-CrossAccount"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAcl",
        "s3:GetObjectTagging",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging",
        "s3:ListBucket",
        "s3:ListBucketVersions"
      ],
      "Resource": [
        "arn:aws:s3:::SOURCE-BUCKET-NAME",
        "arn:aws:s3:::SOURCE-BUCKET-NAME/*"
      ]
    }
  ]
}

2.2. Update inventory/manifest bucket policy

  1. Sign in to the source account Console and open the Amazon S3 console.
  2. Choose inventory/manifest bucket.
  3. Go to the Permissions tab.
  4. In Bucket policy, choose Edit.
  5. Add the following policy to your inventory/manifest bucket by replacing destination account ID and inventory/manifest bucket name:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountManifestAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::DESTINATION-ACCOUNT-ID:role/S3BatchOperationsRole-CrossAccount",
          "arn:aws:iam::DESTINATION-ACCOUNT-ID:user/CONSOLE-USER-CREATING-JOB"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListBucket",
        "s3:ListBucketVersions"
      ],
      "Resource": [
        "arn:aws:s3:::INVENTORY-OR-MANIFEST-BUCKET-NAME",
        "arn:aws:s3:::INVENTORY-OR-MANIFEST-BUCKET-NAME/*"
      ]
    }
  ]
}

For Approach B, the policy also includes permissions for the console user creating the job, because they need to access the manifest file during job creation.

2.3. Update AWS KMS key policy (if using AWS KMS encryption)

This is the critical step often missed in cross-account transfers with encryption. If your source bucket and/or manifest bucket use KMS CMKs, then you must update each AWS KMS key policy to allow the destination account role to decrypt objects.

    1. Sign in to the source account Console and open the AWS KMS console.
    2. Make sure that you are in the right Region. You can change the Region from the top right of the console window.
    3. In the left navigation pane, choose Customer managed keys.
    4. Choose the KMS CMK key being used in source bucket encryption.
    5. In Key policy, choose Edit.
    6. Add the following policy to this AWS KMS policy by replacing destination account ID and the source Region:
{ 
   "Version": "2012-10-17",
   "Id": "cross-account-batch-operations-policy",
   "Statement": [
       {
          "Sid": "AllowCrossAccountBatchOperationsDecrypt",
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::DESTINATION-ACCOUNT-ID:role/S3BatchOperationsRole-CrossAccount" },
             "Action": [
                  "kms:Decrypt",
                  "kms:DescribeKey"
              ],
             "Resource": "*",
             "Condition": {
                 "StringEquals": {
                     "kms:ViaService": "s3.SOURCE-REGION.amazonaws.com" 
                  }
              }
          }
     ]
}

Repeat the same steps for the KMS CMK key being used by the inventory/manifest bucket if that bucket is KMS CMK encrypted.

Important notes for AWS KMS:

  • This policy must be added to the each AWS KMS key being used by the involved buckets at source data bucket account account (source and manifest/inventory bucket).
  • The source bucket and inventory/manifest file bucket exist in a different account than the destination account, thus these buckets must not be using the SSE-KMS AWS managed key (AWS/S3).
  • The kms:ViaService condition makes sure that the key can only be used through Amazon S3.
  • If you have multiple AWS Regions, then include all relevant Amazon S3 service endpoints.

If the destination bucket and/or report bucket use SSE-KMS CMK encryption, then make sure that the destination account batch operations copy roles have appropriate AWS KMS permissions.

Step 3: Create S3 batch operations job

  1. Sign in to the destination account Console and open the Amazon S3 console.
  2. In the left navigation pane, choose Batch Operations.
  3. Create Job:
    • Choose Create job.
    • Choose your AWS region.
    • For manifest format, choose:
      • For Approach A: S3 Inventory report (manifest.json)
      • For Approach B: CSV
    • Specify the path to your manifest file:
      • For Approach A: Path to the manifest.json file
      • For Approach B: Path to your CSV manifest file
    • Choose Next.
  4. Choose operation:
    • Operation type: Copy
    • Destination bucket: Your destination bucket
    • Storage class: Choose appropriate storage class
    • Server-side encryption: Configure as needed for destination bucket
    • Choose other options as applied and choose Next
  5. Configure other options:
    • Completion report:
      • Choose Completion report scope and specify the report bucket where you want to save the job report
    • Permissions:
      • IAM role: Choose the role created in Step 1
    • Choose Next.
  6. Review and create:
    • Review all settings.
    • Choose Submit.
  7. Confirm and run:
    • When the preprocessing completes, the status of job status changes from New to Awaiting your confirmation to run.
    • When the status changes to Awaiting your confirmation to run, choose the job and choose Run job.
    • Monitor progress in the Batch Operations console.

Choosing between approaches

The following table demonstrates the different aspects of each approach:

Factor Approach A (S3 Inventory) Approach B (CSV manifest)
Best for Complete bucket transfers/when scoped data is under known prefixes Selective object transfers
Cost $0.25 per 100M objects API cost to list objects + storage costs
Setup Time Up to 48 hours for first report On-demand, time varies based on number of objects
Automation Fully automated Manual or scripted
Metadata Additional object metadata fields included Basic object info only
Scale Billions of objects Any scale, but manual effort increases
Flexibility All objects in bucket/known prefixes Precise object selection

Troubleshooting common issues

This section offers troubleshooting for the following four common issues.

AWS KMS permission errors

Error: “Access Denied” when copying encrypted objects
Solution: Verify that the AWS KMS key policy includes the destination account role with decrypt permissions. Go to Troubleshooting AWS KMS for more details.

Cross-account access issues

Error: “Access Denied” when accessing the source bucket
Solution: Check that both the bucket policy and IAM role permissions are correctly configured. Review Cross-account access troubleshooting.

Manifest file access

Error: Cannot read inventory manifest or CSV file
Solution:

  • For Approach A: Make sure that the S3 Inventory bucket policy allows access to the destination account role.
  • For Approach B: Verify that the manifest bucket policy includes both the S3 Batch Operations role and console user permissions.
  • Go to S3 Inventory troubleshooting.

CSV manifest format issues

Error: “Invalid manifest format” when using CSV files
Solution:

  • Make sure that the CSV format matches the expected structure (bucket,key or bucket,key,version).
  • Remove any headers from the CSV file.
  • Verify that there are no empty lines or special characters.

Security best practices

  1. Least privilege: Grant only the minimum permissions needed following the IAM best practices.
  2. Condition keys: Use condition keys in policies to restrict access scope.
  3. Monitoring: Enable AWS CloudTrail logging for audit purposes.

Cost optimization tips

  1. Storage classes: Choose appropriate storage classes for destination objects.
  2. Lifecycle policies: Implement lifecycle policies to manage long-term costs.
  3. Inventory frequency: Use weekly instead of daily inventory for cost savings unless necessary.
  4. Regional considerations: Keep source and destination in the same Region when possible to avoid data transfer charges.

Monitoring and reporting

You can access comprehensive reporting through S3 Batch Operations. Go to the tracking job status and completion reports for more details.

  • Job status: You can monitor real-time status updates during execution.
  • Completion reports: Upon completion of the job, you can see detailed reports of successful and failed operations.
  • CloudWatch metrics: You can monitor using Amazon CloudWatch integration.
  • Cost tracking: Track costs associated with batch operations using AWS Cost Explorer.

Cleaning up

After completing your transfer:

  1. Remove cross-account policies: Clean up the bucket and AWS KMS policies in the source account.
  2. Delete IAM roles: Remove the batch operations role if no longer needed.
  3. Archive reports: Move completion reports to cheaper storage classes.
  4. Update inventory: Disable or reduce frequency of inventory reports.
  5. Clean up manifest files: Remove or archive CSV manifest files if no longer needed.

Conclusion

In this post, I showed you how to use Amazon S3 Batch Operations to copy encrypted objects across AWS accounts using two approaches: automated S3 Inventory reports for complete bucket transfers, and custom CSV manifests for selective transfers. I also provided a one-click AWS CloudFormation template that automates the setup with built-in approval workflows.

The CloudFormation template supports both approaches and eliminates the complexity of manual setup while making sure that all security best practices are followed. The built-in approval workflow makes sure that the source account permissions are properly configured before the transfer begins, reducing the likelihood of permission-related failures. For organizations dealing with encrypted data, the AWS KMS policy configuration outlined in this post is essential for successful cross-account transfers. This often-overlooked step makes sure that encrypted objects can be properly accessed and copied across account boundaries.

To get started, try the CloudFormation deployment in your test environment, or explore the S3 Batch Operations documentation for more advanced use cases. If you have questions about this solution, then please leave a comment in the comments section.

Additional resources

TAGS: , , , ,
Abhishek Shukla

Abhishek Shukla

Abhishek is a Senior Technical Account Manager at AWS specializing in data analytics, object storage, and cloud operations. He helps organizations design and implement modernization and migration strategies to AWS Cloud, enabling them to scale efficiently and securely. His work focuses on building resilient architectures that balance performance, cost optimization, and long-term sustainability.

Ezat Karimi

Ezat Karimi

Ezat is a Senior Solutions Architect at AWS, based in Austin, TX. Ezat specializes in designing and delivering modernization solutions and strategies for database applications. Working closely with multiple AWS teams, Ezat helps customers migrate their database workloads to the AWS cloud.