AWS Marketplace
Validate IoT infrastructure with SoftServe IoTForge available in AWS Marketplace
The Internet of Things (IoT) demands a reliable and scalable foundation as device numbers surge, with estimates suggesting 55.7 billion connected devices by 2025. This exponential growth challenges organizations to ensure their IoT solutions can handle large device fleets effectively while maintaining quick response times to user interactions and sensor data. Through AWS Marketplace, a digital catalog that simplifies finding and deploying third-party software on AWS, organizations can access SoftServe IoTForge. This is a comprehensive testing solution designed specifically for these IoT infrastructure challenges, developed by AWS Premier Tier Services Partner SoftServe.
Organizations face several critical challenges in IoT infrastructure testing. Performance bottlenecks, scaling issues, and backend service limitations can significantly impact system reliability, leading to operational inefficiencies and increased costs. With 56% of organizations reporting revenue impacts from technology downtime, comprehensive testing becomes crucial. However, standard testing solutions often struggle with IoT-specific requirements, and building custom solutions requires significant time and specialized expertise.
In this post, we explore how you can use SoftServe IoTForge to conduct thorough IoT infrastructure testing through load, performance, and hybrid testing scenarios. You’ll learn about practical implementations across different use cases, understand SoftServe IoTForge’s architecture on Amazon Web Services (AWS), and discover how its features help identify and resolve performance bottlenecks before they impact operations. We demonstrate how organizations can use SoftServe IoTForge to reliably scale their IoT infrastructure while maintaining optimal performance.
Key testing scenarios
Load and performance testing tools address specific challenges that organizations face. Testing ensures systems work effectively under real-world conditions. Here are three main testing scenarios for IoT infrastructure: load testing, performance testing, and hybrid testing.
Load testing evaluates IoT system performance under increasing concurrent connections and data volumes. Using simulated clients, organizations can validate system behavior across multiple metrics. These include response times and throughput, error rates under various loads, resource usage patterns, and performance benchmarks for optimization.
For example, a smart home products manufacturer discovered critical latency issues through extended load testing of their IoT solution. Ten-hour tests with thousands of Message Queuing Telemetry Transport (MQTT) clients revealed database memory buffer exhaustion, leading to performance degradation. This discovery enabled targeted improvements through database optimization and caching strategies. The following screenshot shows that MQTT PUBACK packets receive latency. After about 4 hours of testing, the latency increased by orders or magnitude.
Figure 1: MQTT PUBACK packets receive latency
Performance testing evaluates system behavior under different conditions to identify optimization opportunities. This includes:
- Stress testing: Identifies system bottlenecks and breaking points under extreme conditions that exceed normal capacity.
- Endurance testing: Evaluates system stability over extended periods to detect memory leaks, resource exhaustion, and performance degradation using consistent load levels.
An organization preparing for market expansion tested their IoT infrastructure to handle double their normal operational load. Gradual increases in MQTT connections and message rates revealed timeout issues in the callback service during message payload validation, allowing for proactive resolution before scaling.
The following screenshot shows test run statistics where request statistics represent measurements captured during the test run. Response statistics show MQTT event latency percentiles.
Figure 2: Test run statistics
The following screenshot shows a test scenario in which clients are gradually added until the target number is reached.
Figure 3: Simulated clients ramp-up chart
The following screenshot shows the publish rate of the MQTT messages during the test.
Figure 4: MQTT messages publish rate during the test
Hybrid testing assesses system behavior under stress across multiple endpoints simultaneously. This approach tests interactions between various system components, reveals issues caused by service dependencies, and provides insights into overall system capabilities and limitations.
A comprehensive test scenario modeled the complete lifecycle of an IoT device, from provisioning through API interactions to data exchange. This approach uncovered a critical delay in propagating MQTT client session information across the broker cluster, leading to cloud-to-device communication failures.
The following screenshot shows the hybrid test statistics in which the success to failure ratio, latencies, and request rate measured for each simulated client type. The fraction of failed requests to different endpoints reached approximately 25 percent of the total number due to simultaneous load on different system components.
Figure 5: Hybrid test statistics
IoTForge: Advanced solutions accelerator for IoT Testing
IoTForge builds on the Locust open source framework for thorough IoT infrastructure testing. It simulates high-scale loads using customizable Python test scenarios. These scenarios can include:
- MQTT interactions with IoT platforms and devices
- HTTP(S) API consumer scenarios
- Support for additional protocols such as SparkPlug B, AMQP, and many others
IoTForge provides real-time statistics through a user interface and generates detailed reports about system behavior under test. This helps organizations identify performance bottlenecks, scaling challenges, and system limitations.
Reference architecture
IoTForge consists of several key components:
- Locust master – Coordinates testing by managing worker nodes, distributing tasks, and aggregating statistics. Provides a centralized UI for monitoring test processes.
- Locust workers – Amazon Elastic Kubernetes Service (Amazon EKS) pods that execute test scenarios. Workers receive instructions from the master, run tests against target systems, and report performance data.
- Amazon ElastiCache – In-memory data store that enables worker synchronization for complex test scenarios.
- Prometheus exporter – Bridges Locust and Prometheus by converting performance metrics for monitoring.
- System entry point – The access point for the system under test. IoTForge simulates client interactions and measures performance under load.
- Prometheus – Time-series data collection system that tracks metrics for both the system under test and IoTForge.
- IoT backend – Applications that process IoT device data, including historians, data lakes, predictive maintenance tools, and analytics services.
The following diagram is the reference architecture.
Figure 6: IoTForge reference architecture
Understanding IoTForge: A technical overview
Understanding how a testing tool works helps you evaluate its capabilities for your needs. IoTForge, built on the Locust framework, shares many core concepts with Locust. Here’s a detailed look at the components and processes that make IoTForge work.
At the foundation are test users, which are digital clients simulating interactions with the IoT system. Each represents specific test scenarios (such as MQTT testing or API testing) and manages test execution.
Test scenarios are Python-based definitions of actions between test users and the system under test. These customizable scenarios can simulate real usage patterns or stress specific components.
Client libraries enable communication between IoTForge’s test users and the IoT infrastructure, supporting various protocols such as MQTT, HTTP(S), Constrained Application Protocol (CoAP), and Advanced Message Queuing Protocol (AMQP).
Custom components are IoTForge-specific elements integrated into the Locust framework, providing advanced features such as distributed test data sharing, additional statistics collection, and resource cleanup.
IoTForge can be deployed in EKS clusters, either in the target system’s AWS account or a separate account. Options include VPC peering for secure communication within the same account, cross-account VPC peering for inter-account communication, and AWS PrivateLink for private connectivity to AWS services such as AWS IoT Core.
Business benefits of IoTForge
IoTForge offers several key benefits. It reduces costs by early identification of issues and performance bottlenecks, minimizing downtime and maintenance expenses. It mitigates risks through proactive discovery of potential weaknesses, preventing service disruptions. The tool enables data-driven optimization by measuring the impact of infrastructure changes and ensuring efficient resource utilization. It validates scalability by verifying that infrastructure can handle increasing device counts and data volumes. Lastly, it enhances user experience by maintaining reliable service delivery through robust infrastructure testing.
Conclusion
As the IoT landscape rapidly expands, organizations must ensure their infrastructure can meet growing demands. This third-party tool addresses these challenges by offering comprehensive testing capabilities tailored for IoT environments. Businesses can use this powerful tool to validate infrastructure readiness, identify performance bottlenecks, and make informed decisions about IoT platform components.
IoTForge’s flexibility allows for precise testing across various protocols and use cases, while its data-driven approach provides actionable insights leading to cost savings and performance improvements. By identifying issues early, companies can avoid costly downtime and maintain a competitive edge in the evolving IoT market.
As we progress towards a more connected future, tools such as IoTForge play a crucial role in establishing the reliability, scalability, and efficiency of IoT infrastructure. Whether expanding IoT offerings or optimizing existing systems, IoTForge offers the capabilities needed to navigate the complexities of the IoT landscape confidently.
Find SoftServe IoTForge in AWS Marketplace to take the next step in strengthening your IoT operations and making your infrastructure ready for future challenges.