Genomics Acceleration, On-prem to the AWS Cloud

Genomics workloads require large quantities of both compute and storage and this rapidly becomes a bottleneck for on prem solutions. Conversely, moving from on-prem data hosting and processing can be a long and drawn-out process, either for a large corporate organisation or a startup and it takes time to scale-up making the “move to cloud” decision difficult. At the same time, your genomics workloads are increasing in size and frequency and an on-prem implementation results in processing delays and inefficiencies for these very large data sets, hampering an organization's ability to deliver research on tight timelines.

However, by working with Zühlke and using this offering, the move to the AWS Cloud can be turned into straightforward task that will give you flexibility, scalability and performance that enables a quick and seamless transition to a highly scalable, high speed and high frequency, genomics processing cluster on AWS.

Methodology and Approach

The Zühlke approach to delivering the production quality PoC in 3-weeks takes a pragmatic operational approach. This is done by “Lifting and Shifting” your current application into a high-performance cluster in the AWS Cloud, to get you to value quickly. A fully documented and production ready system is delivered to the client.

Project Plan

Week 1

• Planning, initiation and lift off workshops
• AWS Infrastructure setup
• AWS Cluster configuration and preparation

Week 2

• Implementation of HPC application code in the AWS Cluster
• Data ingest from on-prem for first genomics workload
• Application and data quality testing

Week 3

• Optimisation of core nonfunctional requirements (NFRs)
• Documentation completion and handover

Outcomes and Deliverables

Implementation of your HPC application code deployed on a high-performance compute cluster using AWS ParallelCluster with SLURM, enabling efficient scaling of computational resources. Amazon Elastic File System (EFS) will be utilized to support a seamless transition from the on-premises NFS storage setup, further enhancing scalability and reducing latency. Amazon S3 will be leveraged for cost-effective storage and lifecycle management. To harness unused AWS compute capacity and reduce costs, on-demand Instances within the AWS ParallelCluster can be used.

Deliverables:

• Operational runbook
• Detailed architecture diagram
• Test results
• Detailed application and system changes walkthrough