NRECA’s cloud transformation: Driving affordability, reliability, and sustainability in the public sector with AWS

This is a guest post from the National Rural Electric Cooperative Association (NRECA), an AWS customer.

From fast-growing suburbs to remote rural communities, America’s nearly 900 electric cooperatives are energy providers and engines of economic growth. Founded in 1942, the National Rural Electric Cooperative Association (NRECA) is a not-for-profit organization that advocates for electric co-op priorities in Washington and helps ensure the long-term success of local co-ops and the communities they serve. It’s a big job. America’s not-for-profit electric co-ops power 56 percent of the American landscape and provide 42 million people reliable, affordable electricity. Additionally, more than 200 electric cooperatives provide broadband service in their communities. As local businesses built by the consumers they serve, electric cooperatives have meaningful ties to rural America and invest $15 billion annually in their communities.

As a part of our focus on affordability, reliability, and sustainability, NRECA recently completed a migration from a traditional on-premises data center and began using Amazon Web Services (AWS) Cloud services. Originally planned for five years, we finished in four. Like most technology, it wasn’t as simple as flipping a switch, but if your organization is considering a similar move, hopefully our experience can act as a guide.

As an IT department, we knew it was essential to use the benefits of modern cloud and software architecture along with eliminating future hardware expenditures in our data center. We also recognized the opportunity to improve the security posture of our mission-critical workloads supporting our member cooperatives. Moving to the cloud offered dynamic scaling and the chance to reduce or eliminate routine tasks in our delivery and operations activities. This approach would help us offer more comprehensive solutions faster and more securely to our business partners.

Types of migrations

We found it was important to gain an understanding of the various types of migrations up front to align with our strategy and give our colleagues patterns for how to get there. AWS has a detailed explanation of the application migration strategies known as the 6 Rs, which were incorporated into our approach. As part of our early research, we discovered the benefits of using managed services and emerging (at the time) serverless technologies. NRECA adopted the 6 Rs and used the strangler fig pattern in many of our migration tasks. For reference, the 6 Rs are:

1. Rehosting – “lift-and-shift”
2. Replatforming – “lift-tinker-and-shift”
3. Repurchasing – Moving to a different product
4. Refactoring / Re-architecting – Thinking differently about architecture, development, and deployments
5. Retiring – Removing the offering from the portfolio
6. Retaining – Leaving it where it is

We started by rehosting and replatforming. Now that our migration is complete, we’re focused on refactoring and re-architecting as our teams mature in their use of AWS.

Preparation

Prior to our cloud migration, the IT department set a goal of improving overall agility. A DevSecOps approach inspired by the book Accelerate by Dr. Nicole Forsgren, Jez Humble, and Gene Kim and the follow-on Accelerate State of DevOps reports published by DORA was adopted knowing this cultural shift was essential from both a business delivery and cloud migration perspective. These tenets influenced our migration philosophy and were summarized to the following for our cloud migration effort:

• Secure by default – Encryption at rest and in transit, authentication and authorization for every endpoint
• Automation everywhere – If it can be automated, it should be
• Continuous improvement – Constant iterations and refactoring of the processes, internal tooling, and frameworks
• Systems over servers – Look at the whole system, don’t just move servers for the sake of moving servers

In conjunction with our strategic cultural and IT refocusing efforts, we needed to prove to ourselves and our business partners moving our applications to the cloud was possible. We identified the first application focused on our cooperative safety program: a typical three-tiered application (UI, API, and database) which was already undergoing a reimagining process with the business partner aimed at aligning the application with newer business processes and updating system technology.

With this in mind, we embarked on architecting the application for AWS from the start. The UI was built on the Angular framework and hosted in Amazon Simple Storage Service (Amazon S3) with Amazon CloudFront for content distribution. Fast response times for our member cooperatives from Hawaii to Maine and Alaska to Florida was a business requirement, so using a content delivery network (CDN) was crucial for pushing content as close to the end user as possible.

Maintaining its on-premises virtual machine roots, the API was deployed to Amazon Elastic Compute Cloud (Amazon EC2) using immutable infrastructure. Amazon EC2 was chosen to make this part of our migration from our virtual machines in the datacenter to AWS faster with fewer initial changes. Because we used immutable infrastructure patterns with AWS CloudFormation and AWS CodeDeploy, we could make our API infrastructure and application deployments automated and consistent. The database migrated to Amazon RDS for SQL Server as its platform, demonstrating the portability of Microsoft SQL from on-premises to AWS for future application migrations.

We used this experience to identify the many dependencies related to hosting, security, operations, and maintenance. This exercise encouraged us to build our initial frameworks for automation and procedures and helped identify where we needed to apply additional focus—leading us to our overall networking and account strategy. Lastly, we pinpointed where to apply resources in future iterations. Early results related to security, cost, and operational efficiency were promising and gave us confidence to proceed with the planning phase for other applications.

After the initial proof of concept (POC) was developed, the team responsible for the POC reviewed their application portfolio for cloud readiness by cataloging in terms of the types of migration, target AWS services, technologies, dependencies, and any software updates required. This team was best positioned due to their knowledge and experience derived from the initial POC. Next, the IT department prioritized speed over perfection to enable complete migration in five years. This aligned all application teams to initially use a combination of rehosting and replatforming with an eye on long-term refactoring and re-architecting.

Early emphasis was placed on obtaining AWS certifications. Each team was required to have at least one member hold a Cloud Practitioner certification as a minimum to get started. This helped ensure AWS terminology was commonly understood. When we mentioned S3, EC2, or RDS, time was not wasted on explaining foundational concepts. Many teams extended their education to more advanced certifications over the course of the migration journey. In some cases, teams were augmented with new staff members or contractors who already held certifications. By the end, approximately 65 percent of staff had some level of AWS certification. Don’t neglect this step on your cloud migration journey because speaking a common language reduces the chances of business goals being lost in translation when applying them to the value proposition of the cloud.

Execution

Although our guiding approach was to look at systems over services, the best metric to measure our progress was to track the individual physical and virtual server retirements in our on-premises environment. Initially, we used spreadsheets to track our progress. As we matured, our IT service management (ITSM) team created dashboards on top of our configuration management database (CMDB) to give everyone a high-level status view. Each team was responsible for planning their own portion of the migration because they’re closest to the actual work.

Because much of the migration was cross-cutting (security, network, architecture, and technology stack), the teams communicated their plans and progress during a weekly cloud migration status meeting. Our cloud architecture and enablement (CAE) team provided guidance, governance, and assistance regularly to the teams during their active migration phases. These regular check-ins allowed the CAE team to step in to help remove blockers and kept teams accountable for their progress while augmenting our InnerSource repositories and maturing our cloud migration checklist.

There are two big, cataclysmic events which threaten to derail or even defeat a cloud migration effort. Those are misconfigurations leading to data breaches or leaks and cost overruns. To address misconfigurations, our security team implemented both service control policies (SCPs) within AWS Organizations and third-party cloud security posture Management (CSPM) tools. The CSPM tools monitor and detect misconfigurations based on policies centered on NIST, Health Insurance Portability and Accountability Act (HIPAA), Payment Card Industry Data Security Standard (PCI DSS), and other common frameworks. Through these tools, our cloud security team actively monitors the organization’s cloud security posture and reaches out to teams immediately when a misconfiguration is identified.

To address the cost overrun concern, we enforced the use of tags through tag policies and implemented a FinOps practice with active monitoring using a third-party cloud cost management and optimization tool. As a part of the active monitoring, our FinOps lead routinely meets with teams to discuss how they can reduce costs and rightsize as a part of their daily operations. NRECA realized a 30 percent reduction in cost during the migration as compared to our on-premises costs. Through a combination of Enterprise Discount and Savings Plans and FinOps check-ins focused on workload optimization, NRECA has realized an additional 30 percent savings over our initial costs. Optimization activities include serverless adoption and turning off workloads during nonbusiness hours. As a part of our enterprise agreement, we periodically engage with our AWS Nonprofit team to do a cost evaluation. This evaluation has shown our workloads to be better than 95 percent cost-efficient for their size and operation.

Although it’s tempting to think all workloads will migrate to AWS, set the expectation that some workloads aren’t suited for the cloud. There are many reasons a workload isn’t appropriate, such as legacy software and systems, which might be retiring or replatforming within a certain window where the effort to migrate doesn’t make financial sense or fit priorities.

What’s next?

Here at NRECA, we’re hyper focused on refactoring and re-architecting. One of the teams which rehosted their systems midway through the migration is now embarking on a complete redesign focused on event-driven architecture with entirely serverless components. Other teams with three-tiered applications are moving away from EC2 for their APIs and are adopting Amazon API Gateway with AWS Lambda. These efforts are motivated by the cooperative principles of affordability, reliability, and sustainability and help us meet our mission while moving away from routine and repeated technical tasks to offer our customers greater value.

Getting started

Focus on your people and culture. Do you have core individuals properly educated and trained? Is the organization ready to make the shift? Define your end goal and work backwards to where you are today. More tactically, start small by picking a nearly cloud- ready application and migrate it first. Learn from the process, iterate and scale, find opportunities to refactor and build InnerSource libraries and a cloud migration checklist. Monitor your costs by using a FinOps tool, either third-party or the AWS built-in cost management capabilities. Lastly, plan and be flexible. Conditions and situations change so be ready to adjust your plans along the way.