Encored Technologies Successfully Built an HPC on AWS for Weather Research & Forecasting (WRF)

Encored Technologies provides a comprehensive solution for VPP sellers ranging from power generation prediction, floor control, and market bid optimization. It focuses on helping energy sellers create stable revenue in the VPP market by adopting AI-based technologies for power generation prediction and optimization.

Encored Technologies built an in-house weather forecasting system to improve the accuracy of power generation from renewables such as solar and wind.

To generate weather forecasts, Encored Technologies carries out four calculations a day, with each calculation taking up to two hours using a cluster of 500 cores (maximum 4,000 cores daily), generating approximately 500GB of output every day. The company loads and processes about 100TB of data on one go while creating the training data for its power generation forecasting AI model.

Encored Technologies chose AWS’ HPC solution to save HPC environment configuration time and operate HPC clusters in a cost-efficient manner. EVP Hyoseop Lee says building an on-premise datacenter with massive computing power usually takes at least 6 months plus the cost of configuration and operation, but it took only three days with AWS.

Encored Technologies built its initial HPC environment by creating an HPC cluster for weather forecasting using Amazon EC2 Hpc6a, an x86-based Amazon EC2 instance type for HPC. It stored the big data required for weather forecasting using Amazon Simple Storage Service (Amazon S3) and Amazon FSx for Lustre. It used AWS ParallelCluster, an open source cluster management tool, for easy deployment and management. This allowed the company to accelerate the process of building a weather forecasting system and boost operational efficiency. Encored Technologies saved cost by 60 percent after changing the instance to Amazon EC2 Hpc7g which is an ARM-based, HPC-dedicated instance type powered by AWS Graviton 3E processor.

Encored Technologies is producing weather forecasts combining multiple weather variables. Based on this, it performs calculations using up to about 4000 cores to predict power generation - a task only a high-end HPC cluster or a supercomputer could do in the past.

EVP Hyoseop Lee explained, “Nobody was sure if building the same system on an on-premise environment can give us the performance we want. Uncertainty prevents us from adopting the system in the first place, but AWS removed that uncertainty for us. We couldn’t have imagined predicting the weather if it were not for AWS’ HPC solution.” He added, “The PoC allowed us to build the hardware resource required for testing within only a day. Our researchers can directly configure the necessary HPC cluster environment as a code, which means we can remove or change clusters whenever we want. That in turn eliminated the need for having a dedicated infrastructure management staff.”

Lee further explained, “AWS-based HPC environment helps us save cost and improve performance. We are planning to optimize cost by leveraging various purchase options in AWS including Reserved Instance and Spot Instance. We look forward to working with AWS as we plan to go global with our energy AI service.”

To learn more, visit Encored Technologies.