AWS Storage Blog
Category: AWS CloudShell
Scalable cross-cloud data migration to Amazon S3 with distributed rclone
Migrating petabytes of data across cloud providers is one of the most operationally demanding tasks an organization can take on. At this scale, simple transfer approaches break down. Teams lose track of what has been copied and what has failed. Transfers stall and require constant manual intervention to restart. In some cases, teams need to […]
Implement single-exchange tokens for short-lived Amazon S3 presigned URLs with Terraform
Organizations across industries use signed URLs to grant temporary, credential-less access to private resources such as receipts, medical or financial records, legal files, or confidential reports. However, signed URLs can be reused by anyone until they expire, creating security risks if a URL is shared or inadvertently disclosed. This risk can be mitigated by vending […]
Accelerating Amazon S3 Batch Operations at scale with on-demand manifest generation
Modern enterprises routinely manage billions of objects across their cloud storage environments, needing efficient bulk operations for disaster recovery, compliance management, data transfer, and cost optimization. Performing these operations manually or through custom scripts becomes impractical at scale, often creating operational bottlenecks when time-sensitive actions are necessary. Organizations frequently need to identify and process specific […]
Detect malware threats using AWS Transfer Family
Securely sharing files over SFTP, FTP, and FTPS is a staple within many business-to-business (B2B) workflows. Across industries, companies use file transfer to transmit inventory, invoice, and compliance information. It is critical for companies to make sure that shared files do not have any malicious content that could compromise their systems. Guaranteeing the shared files […]
Encrypt and decrypt files with PGP and AWS Transfer Family
1/11/2024: Updates made due to CloudShell migration to Amazon Linux 2023 (AL2023). Protecting sensitive data is not a novel idea. Customers in industries like financial services and healthcare regularly exchange files containing sensitive data, including Personal Identifiable Information (PII) and financial records with their users. Pretty Good Privacy (PGP) encryption of these files is often […]
Use AWS Backup and CI/CD tools to automate centralized backup across AWS services
Automating and scaling your data protection and backup strategy helps you reduce manual overhead from time-consuming configuration, minimizes the risk for errors, provides visibility on drift detection, and enhances backup policy compliance across distributed AWS workloads or accounts. Incorporating backup in your disaster recovery (DR) and business continuity plan (BCP), along with the automation of […]
Automate centralized backup at scale across AWS services using AWS Backup
UPDATE 11/11/2021: Updated to improve the user experience in the walkthrough steps. Reliable and consistent backup of cloud data is important to have a secure data archive and restore in the event of data loss. Traditionally, backup administrators used manual backup procedures that were difficult to scale and lowered productivity, with data distributed across multiple […]