Volvo Cars Streamlines In-Vehicle Software Testing with AWS Graviton on Amazon EKS

Volvo Cars Superset tech stack

Software is replacing hardware as the primary driver of innovation and value creation. It is ushering in a new era of mobility, where vehicles can integrate seamlessly into society and be rolled out globally.

For Volvo Cars, the software-defined vehicle, realized with support of Amazon EKS and AWS Graviton, holds a particularly transformational power – the ability to further raise the bar on safety and to improve quality of life for its customers. To capitalise on this, Volvo Cars recently introduced a new “game-changing” approach to technology, Volvo Cars Superset tech stack.

As the Superset tech stack underpins all Volvo Cars’ upcoming vehicles, it converges all engineering efforts into one single direction, rather than working on specific, siloed, car projects. This means Volvo Cars can continuously enhance the performance of each vehicle in its lineup simultaneously, so that EX90 customers benefit from ES90 software upgrades and vice versa.

This “closed-loop” development approach, leveraging data, connectivity, software, and core computing, will enable Volvo Cars to constantly improve every aspect of its cars over time.

In this blog post, you will learn how Volvo Cars has built a scalable Continuous Integration (CI) platform on Amazon Elastic Kubernetes Service (Amazon EKS) and AWS Graviton supporting the Superset tech stack. CI is leveraging tools like Zuul, Gerrit, Buildbarn, Bazel, and JFrog Artifactory to enable efficient testing and deployment of in-vehicle software. Additionally, you will follow Volvo Cars’ journey in building this platform demonstrating how cloud-based automation and tooling can streamline complex automotive software development on AWS.

Deep Dive Superset tech stack

Volvo Cars’ Superset tech stack includes following fundamental platform stacks:

1. A powerful core computing system that enables a closed-loop development process, built on NVIDIA Drive Orin SoC, capable of over 250 trillions of operations per second (TOPS), and Qualcomm Technologies’ Snapdragon Cockpit Platforms. These orchestrate everything in the car and power the deep learning capabilities underpinning AI-based active safety and driving assistance systems.
2. 5G connectivity and integration with Volvo Cars’ cloud infrastructure for Connected Car Cloud (C3) built on AWS.
3. A data center with advanced AI capabilities to further enhance safety features and is set to be the largest AI training center in the Nordics.
4. Ecosystem integration to allow the cars seamlessly integrate with drivers’ digital lives built on multiple operating systems and computers in the car to continuously learn from real-world data and improve the cars over time.

Embedded software tailored for each electronic control unit (ECU), the QNX core computing platform and the NVIDIA Orin SoC are part of the core computing system (1) within the Superset tech stack. These software components undergo extensive integration testing over months of development. A critical element of this process is the CI (Continuous Integration) platform.

Figure 1 Scalable Product Architecture (SPA3), the electric technology base underpinning Volvo Cars Superset tech stack, is a Zonal Architecture with 2 core computers, 2 zonal controllers and multiple ECUs

Impediments while developing the Superset tech stack and Strategies for Overcoming Them

Expensive Hardware-in-the-Loop (HiL) tests
To test, verify and be automotive standard compliant for the multitude of software running on Superset tech stack, Volvo Cars has opened a new, state-of-the-art software testing center in Gothenburg with an initial investment of around 300 million SEK. This 22,000 square meter facility will employ 300 people once at full capacity and house around 500 testing rigs and digital test environments. While greatly expanding Volvo Cars’ capacity for integrated software testing across all levels, the test facility will incur significant costs for Volvo Cars.

HiL tests are a bottleneck
The company’s current Hardware-in-the-Loop (HIL) testing rigs with NVIDIA and Qualcomm compute nodes are a scarce resource. Testing on these rigs is creating a significant bottleneck in the development workflow considering that the development teams issue over 6,000 software changes each day resulting in 35,000 builds per day which in turn requires around 250 tests on HiL rigs every single day where the individual tests must continuously wait on HiL queues.

Solution is HiL independence by virtualization
Volvo Cars is shifting-left towards a cloud-based, virtualized approach for testing its software-defined vehicle (SDV) architecture on the AWS platform, which enables more cost-effective and scalable testing without relying on as much physical hardware and test facilities. Virtualizing the Electronic Control Units (ECUs) and performing Software-in-the-Loop (SiL) testing in a cloud-based environment allows Volvo Cars to run multitude of software verification tests to improve the overall quality and stability of its in-vehicle software, and it also helps identify and address integration issues more easily in a virtualized environment compared to HiL rigs, ultimately driving down the overall costs involved in validating and refining Volvo Cars’ next-generation SDV technologies.

Technology mismatch in virtualization and testing
The SiL tests were previously running on a Linux emulation environment that lacked many critical aspects of the target software and hardware. There was a runtime environment gap in the CI-process, where the emulation lacked the QNX runtime Volvo Cars uses and moreover had incorrect target CPU architectures (AMD vs. ARM), leading to excessive recompilation efforts and errors when tests reached the costly HiL environments.

There was a clear need to run the target binaries in the cloud with minimal changes on virtual compute instances and matching ARM-based target CPU architecture in vehicle.

Solution is Environmental parity
To scale the SiL tests of in-vehicle software, Volvo Cars superset tech stack development teams built a CI-platform running on Amazon EKS. The embedded software tested is specifically targeting Qualcomm Snapdragon chipsets. To ensure the environmental parity, Volvo Cars utilizes ARM-based AWS Graviton instances and QNX-provided Amazon Machine Images (AMIs) for QNX Neutrino RTOS 7.1 from the AWS Marketplace and partners with Synopsys for virtualizing nodes at the Zonal ECUs.

Volvo Cars decided that t4g.2xlarge instances are the most cost-effective and appropriate ones for this use case and opted for AWS Graviton 2 processors. The core count and clock speeds of these instances are best aligned with the specifications of the hardware in the cars. Volvo Cars is considering whether to adapt AWS Graviton 3 processors for testing future car models with higher specifications.

Volvo Cars has in the recent model program introduced dual NVIDIA Drive Orin configuration in the Volvo Cars ES90 model making it the most powerful car they have ever created in terms of core computing capacity. Recently the CI-pipelines and AWS Graviton nodes have been adapted to support testing software targeting NVIDIA DRIVE Orin configurations as well.

Complex software dependencies
In automotive software development, multiple teams, often from different companies, collaborate using various cloud platforms and version control systems like GitHub, GitLab, Gerrit, and Pagure. These teams work on interdependent components such as ADAS, connectivity, and scene detection, all of which must integrate into a single deployable package. Managing changes across separate repositories is complex, especially when updates in one depend on others. Integration issues often arise if changes aren’t tested together early.

To address this, it’s critical to detect breaking commits and failed integrations before code is merged—across projects, repositories, and organizations. This proactive approach streamlines development, reduces failures, and improves software quality.

Solution is Project Gating
To solve integration issues early, the concept of Project Gating was introduced. It blocks untested or incompatible changes, across repositories from being merged. The CI pipeline only allows changes that pass tests against all pending modifications, ensuring stable integration before merging.

In contrast, traditional development relies on developers to test locally or commit changes directly, with CI catching issues afterward. This reactive approach is error-prone and can lead to failures during CI, in HiL tests, or worse—in the final product.

Figure 2 shows gating approach in operation which stops a faulty commit(c3) to be merged and would allow only changes (c1,c2,c4) to be merged if they pass the tests in combination even though the changes are scattered across different repositories.

These interdependencies can even get more complicated when there are dependencies in the zones of the vehicle architecture. Following figure illustrates the interdependencies extracted from the CI orchestrator Zuul, chosen as the platform for supporting “Project-Gating” processes by Volvo Cars.

Figure 3 Visualizing an example CI-pipeline in Zuul orchestrator showing the dependancies, with dotted lines, to which zonal ECU or Core compute the software packages under test are targeting within the vehicle electrical architecture

The CI Platform
Let’s examine the key components of the CI-platform and the distinct functionalities each component fulfills.

Figure 4 CI Platform architecture and components on AWS

1. Over 2,000 embedded software developers commit code changes to over 1,000 GitHub repositories and collaborate on Gerrit, occasionally creating conflicting changes which can disturb the integration tests.
2. Zuul-CI orchestrates these tests defined on Jenkins and enforces project gating by simulating the final merges.
3. The resulting build artifacts and any intermediate artifacts are stored in JFrog Artifactory.
4. Volvo Cars’ CI platform runs over more than 35,000 software tests daily, involving the frequent download and upload of source code, test definitions, and binary packages, including large base images on Artifactory, resulting in unnecessary traffic with added complexity and costs; to address this, Volvo Cars is using Buildbarn and Bazel. These tools enable caching, remote execution of build actions and parallel build execution, thereby speeding up build times by offloading computation to remote servers and reducing the overhead and duplications in the build processes.
5. Some of the continuous integration steps involve executing tests on HiLs located on-premises. Volvo Cars has established dedicated network connections with AWS Direct Connect to their test facilities.
6. After the software passes all the validation steps and verified, it is finally released by triggering the release pipeline defined on Jenkins. Following additional compliance and security steps, the final package is sent to vehicles through Over-The-Air-Update utilizing Volvo Cars’ Connected Car Platform (C3). C3 also runs on AWS, coordinating vehicle communication and data exchange with the vehicles.
7. The CI platform is built on Amazon EKS and AWS Graviton. Volvo Cars’ primary objective is to make it the standard and central CI tool across their organization. This objective requires full autoscaling on both pod and node level, which is facilitated by the use of Karpenter. Previously teams were utilizing static autoscaling. Grafana and Splunk alarms were used to manually monitor the system and whenever needed new nodes were added by hand where an additional booting step was also required. Instances took up to a minute to launch. Usually, the nodes were larger than necessary and was underutilized most of the time. By leveraging Karpenter Volvo Cars can now enjoy more flexible node provisioning to automatically scale across a diverse range of node types. Based on resource request from the pods, Karpenter will automatically choose the most cost-effective nodes based on that particular resource request. In this new setup less monitoring is required as Karpenter is monitoring the system on behalf of the team and it takes seconds to launch nodes as no manual effort is needed in the node scaling process. Volvo Cars adapted Karpenter even for network intensive applications using custom metrics.

AWS is enabler of Superset tech stack

The applications that are part of the CI platform were previously deployed across multiple cloud platforms. This distributed approach created communication bottlenecks and limited scalability. Adapting AWS recommendations, Volvo Cars decided to consolidate the infrastructure onto Amazon EKS within a single region and multiple availability zones. This consolidation effort helped Volvo Cars reach over 35,000 test and build jobs on the CI-platform every day, demonstrating significant performance improvements over the previous multi-cloud setup which supported at most 5,000 jobs per day.

Figure 5 Daily pattern of number of completed check, gate and release jobs in Volvo Cars Zuul tenant earlier this year

The improvements that increased overall performance of the CI pipelines for software builds and tests enabled running 7 times more tests before the in-vehicle software is released, with automated identification of software issues and as a result help Volvo Cars roll out more stable in-vehicle software.

Conclusion

In this blog post we walked you through the scalable Continuous Integration (CI) platform Volvo Cars built on Amazon EKS to manage the Software-in-the-Loop (SiL) testing for their in-vehicle software development. We described in detail the components of the platform and their functions enabling Volvo Cars to process over 35,000 test and build jobs daily with significant improvements compared to their previous multi-cloud setup.

Please get in touch with AWS Solution Architecture teams to learn more about how Amazon EKS, AWS Graviton together with Karpenter can help scale similar CI platforms to your needs.