Cost-efficient backup archiving with Veeam Direct to Amazon S3 Glacier storage classes

If you work with data storage and data protection, then you’re aware of the “3-2-1 rule.” System administrators consider the 3-2-1 rule a best practice for backup and disaster recovery (DR), and it is recommended by US-CERT. The 3-2-1 rule states that you should have three copies of your data (your production data and two backup copies) on two different types of media with one copy off-site for DR. Veeam users looking to implement the 3-2-1 rule with a significant footprint in an on-premises performance tier and that need very low Recovery Time Objective (RTO) for large amounts of data are constrained by the amount of data they can recover over a network connection. However, many are interested in using cloud as an archive destination, not as a location for their Veeam performance tier or capacity tier.

Today, Veeam Backup and Recovery (VBR) transfers backups to Amazon S3 object storage and refactors those backups for optimal storage in Amazon S3 Glacier storage classes. This minimizes archiving cost by storing large objects with long retention times (storing data in S3 Glacier Instant Retrieval can be 68% lower cost than storing in S3 Standard-Infrequent Access). Prior to VBR 12.2, users with a large on-premises object storage footprint (driven by RTO) needed to create a duplicate copy of backups in S3 Standard before moving backups to S3 Glacier storage classes. VBR 12.2 has removed this requirement with its Direct to Glacier feature. You can now offload backup archives (marked with GFS flags for long-term storage) from on-premises storage devices directly into S3 Glacier storage classes, without landing in S3 Standard first.

In this post, I walk through the process of configuring S3 Glacier storage classes as part of an existing VBR configuration, without needing an intermediate copy on Amazon S3 Standard storage classes. This allows users with obsolete offsite archival mechanisms to use cloud object archive storage, improve resiliency, and reduce the access time for archived data.

Amazon S3 Glacier storage classes

Hybrid cloud users have depended on Amazon S3 since its launch in 2006 as a safe landing zone for both primary data and at least one of the backup copies stipulated by the 3-2-1 rule. Today Amazon S3 holds more than 400 trillion objects, and supports the 3-2-1 rule for many Veeam users. Cloud adoption has grown in part because of its resilience. Amazon S3 is designed to provide 99.999999999% durability. As a result, users have decreased their dependency on costly, inflexible on-premises backup storage appliances. Instead, they now use Amazon S3 Standard as the location for the first backup copy and Amazon S3 Glacier storage classes as the location for long-term archives. S3 Glacier storage classes are designed to be long-term, secure, and durable storage for data archiving. The three S3 Glacier storage classes are available to VBR users: Amazon S3 Glacier Instant Retrieval, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive . The three storage classes are each optimized for different access patterns and usage, with Glacier Instant Retrieval offering retrieval times in milliseconds, Glacier Flexible Retrieval offering retrieval times in minutes (when batch operations are enabled in VBR), and Glacier Deep Archive offering the lowest-cost archive with retrieval times within 12 hours.

Archiving Veeam backups

VBR migrates backups marked for long-term retention to archive storage. S3 Glacier Instant Retrieval and S3 Glacier Flexible Retrieval storage classes have a 90-day minimum retention, and S3 Glacier Deep Archive has 180 days minimum (which means objects deleted before the retention period still incur charges). Therefore, archive storage does not make sense for daily backups with a short retention period. VBR doesn’t move backups if their retention period is lower than the minimum for the class. VBR also refactors data from smaller object sizes into larger archive objects when moving a backup to archive. This is done to cost-optimize consumption of S3 Glacier storage classes and API calls. This refactoring typically takes place on the VBR proxy itself or a gateway, if one is configured. You can find more details in the Veeam documentation.

Getting started with Veeam Direct to Glacier

To add a Scale-Out Backup Repository (SOBR) with Direct to Glacier archive tier, choose Backup Infrastructure → Scale-out Repositories in the VBR console, then choose Add Scale-out Repository, as shown in Figure 1.

Figure 1: New Scale-out Backup Repository

Choose the Performance Tier and the Placement Policy. Prior to VBR 12.2, choice of Capacity Tier was necessary. As of VBR 12.2, you can omit the checkbox on the Capacity Tier page, and proceed to Archive Tier. On the Archive Tier page, check the Archive GFS full backups to object storage box and choose Add, then choose Amazon S3 Glacier, give your new archive repository a name, and add it using credentials as you would any other repository, as shown in Figure 2.

Figure 2: Adding S3 Glacier Repository

Note the Archive GFS backups older than X days selector. Backups are not moved to archive until they are older than the specified number of days.

First specify credentials for the AWS Account to use, then choose the appropriate bucket. Next, configure your desired storage class, as shown in Figure 3. You can find general information on adding AWS storage for Veeam here.

Figure 3: S3 Glacier storage class configuration in VBR

Decide whether you configure an Archive helper appliance. In most cases, users choosing Direct to Glacier use a local, disk-based performance tier, in which case a helper appliance is not used.

Figure 4 shows a SOBR configured with a local tier and an archive tier on S3 Glacier storage classes:

Figure 4: VBR S3 Glacier storage classes SOBR

S3 Object Lock for S3 Glacier storage class archives

The prevalence of ransomware has resulted in an extension to the 3-2-1 rule, now referred to as the “3-2-1-1-0 rule.” The term reflects the requirement to keep one copy of data in an immutable or WORM repository, and to regularly restore backups to test and confirm zero errors. To enable S3 Object Lock on an archive repository, first enable S3 Versioning on the S3 bucket (and do not select a default retention mode or duration), then enable S3 Object Lock on the bucket. This allows you to check the Make backups immutable box when creating a new archive repository in VBR, as shown in Figure 5.

Figure 5: Enable S3 Versioning and S3 Object Lock on S3 bucket

Enabling S3 Object Lock in Veeam necessitates checking a box, as in Figure 6.

Figure 6: Enabling S3 Object Lock on an archive repository

Enabling S3 Object Lock protects your data even if a bad actor compromises your backup server or gains access to bucket credentials. Objects can’t be deleted or modified within the retention period.

Cleaning Up

Don’t enable immutability on a test bucket (or you won’t be able to delete test data until the immutability period expires), and delete any test buckets after backup and restore tests are complete to avoid ongoing charges.

Conclusion

In this post, I covered the process of configuring S3 Glacier storage classes as part of an existing VBR configuration, without needing an intermediate copy on Amazon S3 storage. This is a powerful new feature of Veeam Backup and Replication 12.2.

It allows users to avoid costly upgrades to on-premises storage as it nears capacity limits. This is done by using the virtually unlimited scalability of Amazon S3 for 68% lower storage cost. It also enables the benefits of long-term archival storage for users who can’t use Amazon S3 as a capacity tier for VBR. Find more information on VBR here.

I encourage VBR users with long-term archive requirements to create an S3 bucket, and a VBR repository with an S3 Glacier storage class, to try this new functionality.