Efficient video streaming and vision AI at the edge with Realtek, Plumerai, and Amazon Kinesis Video Streams

Artificial Intelligence (AI) at the edge is popular among smart video devices. For example, Smart Home cameras and video doorbells revolutionized home monitoring. What began as simple recording and remote viewing tools has evolved into intelligent observers. With AI infusion, today’s cameras can actively analyze scenes, alert users to motion events, recognize familiar faces, spot package deliveries, and dynamically adjust their recording behavior. Enterprise surveillance cameras are another example. These cameras have superior resolution, enhanced computing power, and can drive more sophisticated AI models. These enhanced capabilities result in sharper detection at greater distances.

As illustrated, customers demand intelligent monitoring systems that can process data locally while maintaining privacy and reducing bandwidth costs. To address these needs, the AWS Internet of Things (AWS IoT) team has developed a smart camera solution with AWS partners that combines Amazon Kinesis Video Streams, Realtek’s low-power Ameba Pro2 microcontroller, and efficient machine learning models from Plumerai. This blog post provides guidance for event-triggered video uploads coupled with human detection algorithm processing at the edge.

Solution architecture

Figure below illustrates the solution architecture that this blog uses:

1. Beginning with the camera, the device firmware has integrated Realtek SDK to access camera modules via defined APIs.
2. The video fragments are delivered to Plumerai’s machine learning models for object detection.
3. The sample application adds detection results as bonding box overlay on the original video fragments. This sample continuously uploads the fragments to cloud through Kinesis Video Streams Producer SDK. (As an aside, you can also set detection results to trigger uploads of 20-second video segments.)
4. The Kinesis Video Streams Producer SDK relies on PutMedia API with long HTTPS connection to upload MKV fragments continuously in a streaming way.
5. The media data will be ingested and the service stores all media data persistently for later analysis.
6. A frontend application performs the playback of live, or previously recorded videos, relying on HLS or DASH protocols from Kinesis Video Streams.
7. The solution feeds video and audio data into Large Language Models (LLMs) for Agentic AI insights. (We will cover semantic video search in our next blog).

Integration highlights

Amazon Kinesis Video Streams

Kinesis Video Streams transforms how businesses handle video solutions for IP cameras, robots, and automobiles. Key benefits include:

• A fully managed architecture. This helps engineering teams focus on innovation instead of infrastructure and is ideal for companies with limited resources.
• AWS SDKs are open-sourced. Top brands especially value this independence from platform constraints.
• Flexible pay-per-use pricing model. While device development can take months or years, you don’t pay until the cameras go live. With typical cloud storage activation below 30% and declining yearly usage, costs stay dramatically lower than fixed license fees.

Plumerai

The Plumerai company specializes in embedded AI solutions, specially focused toward making deep learning tiny and efficient. The Plumerai model helps to provide inference on small, affordable, and low-power hardware. The company also optimizes AI models for the Realtek Ameba Pro2 platform through:

• Assembly-level optimizations can maximize Arm Cortex-M CPU performance, and leverages DSP instructions for enhanced signal processing capabilities.
• Neural Architecture Search (NAS) selects optimal AI models for Realtek NPU and memory architecture to achieve 0.4 TOPS NPU acceleration
• Plumerai models use Realtek on-chip hardware accelerators (scalers, format converters) to reduce computational load.
• The AI model supports RTOS to seamlessly integrates the SoC’s real-time operating system.
• The application integrates with Realtek’s media streaming framework.
• The fast boot design supports rapid booting times, which improves battery life, and ensures high speed of active object detection.
• The edge AI models are trained on 30 million labeled images and videos.

These enhancements translate into the following real-world performance:

• Delivers precision without wasting memory.
• Captures wide scenes through 180° field-of-view lenses.
• Detects individuals at 20m+ (65ft) distances.
• Handles crowds by tracking 20 people simultaneously.
• Maintains individual tracking with a unique ID system.
• Performs consistently in bright daylight and total darkness.

Realtek Ameba Pro2

Figure above illustrates Realtek Ameba Pro2’s data architecture. It contains Integrated Video Encoder (IVE) and an Image Signal Processor (ISP) that processes media’s raw data and delivers the result to a Video Offload Engine (VOE). The VOE then manages multiple video channels and concurrent video streams to support the motion detection algorithm. The Neural Processing Unit (NPU) performs inference on images or image regions. The Parallel Processing Unit (PPU) handles multitasking jobs like cropping Regions of Interests (ROIs) from high-resolution images, resizing NPU inference input, and retrieving final output from high-resolution channels.This architecture unlocks powerful capabilities to support video analytics at the edge, including:

• Runing with minimal CPU power for maximum efficiency.
• Responding in near real time to motion.
• Begin video processing even during the boot sequence.
• Streaming to both the SD card and cloud through secure WiFi or Ethernet.
• Leveraging NPU to deliver superior AI performance.
• Integrating with Plumerai models and Kinesis Video Streams through a multimedia framework SDK.

Walkthrough

This section outlines the building steps for the solution to run edge AI and stream the video fragments.

Prerequisites

• AWS account with permission for:
  • AWS console login
  • Kinesis Video Streams API (GetDataEndpoint, DescribeStream, PutMedia)
  • Amazon Elastic Compute Cloud (Amazon EC2) instance creation to build SDKs and binaries
• A stream resource with the name “kvs-plumerai-realtek-stream” created on Kinesis Video Streams Console.
• The Realtek Ameba Pro2 Mini MCU.
• Basic knowledge about embedded systems and working in a Linux environment.
• Internet connection to download the SDK and upload videos to AWS.
• Library and machine learning model files from Plumerai. (Please submit your request on the Plumerai Website.)

Set up the building environment

This blog uses an Amazon EC2 with Ubuntu LTS 22.04 as the building environment. You can use your own Ubuntu computer to cross-compile the SDK.

Amazon EC2 instance setup:

1. Sign in into the AWS management console and navigate to Amazon EC2.
2. Launch an instance with the following configuration:

• Instance name: KVS_AmebaPlumerAI_poc
• Application and OS Images: Ubuntu Server 22.04 LTS (HVM)
• Instance type: t3.large
• Create a new key pair for login: kvs-plumerai-realtek-keypair
• Configure storage: 100GiB
• Follow SSH connection prerequisites to allow inbound SSH traffic.

Download sample script from Github:

1. Using the following command, log into your Amazon EC2 instance (be sure to replace xxx.yyy.zzz with the instance’s IP address). For detailed instructions, see Connect to your Linux instance using an SSH client.

ssh -o ServerAliveInterval=60 -i kvs-plumerai-realtek-keypair.pem ubuntu@54.xxx.yyy.zzz
git clone https://github.com/aws-samples/sample-kvs-edge_ai-video-streaming-solution.git
cd ./sample-kvs-edge_ai-video-streaming-solution/KVS_Ameba_Plumerai

Obtain the Plumerai library:

2. Using the Plumerai contact us form, submit a request to receive a copy of their demo package. Once you have the package, replace the “plumerai” directory with it in the Amazon EC2 instance. The updated directory structure should be the following:

Obtain the Ameba SDK:

3. Please contact Realtek to obtain the latest Ameba Pro2 SDK. In the directory structure, replace the “ambpro2_sdk” in Amazon EC2. The directory structure should be the following:

Install dependencies and configure environment

4. Run the script setup_kvs_ameba_plumerai.sh in the directory sample-kvs-edge_ai-video-streaming-solution from the Github repository:

chmod +x setup_kvs_ameba_plumerai.sh./setup_kvs_ameba_plumerai.sh

The script will automatically install the Linux dependencies, build the Realtek toolchain, run necessary Plumerai patches, copy model files, and download the Kinesis Video Streams Producer SDK. If you experience an error in the process, please contact Realtek or Plumerai for technical support.

Configure sample in Kinesis Video Streams Producer SDK

Use the following to configure AWS credentials, stream name, and AWS region. These can be found in the component/example/kvs_producer_mmf/sample_config.h file.

// KVS general configuration
#define AWS_ACCESS_KEY "xxxxx"
#define AWS_SECRET_KEY "xxxxx"
// KVS stream configuration
#define KVS_STREAM_NAME "kvs-plumerai-realtek-stream"
#define AWS_KVS_REGION "us-east-1"
#define AWS_KVS_SERVICE "kinesisvideo"
#define AWS_KVS_HOST AWS_KVS_SERVICE "." AWS_KVS_REGION ".amazonaws.com"

Comment out example_kvs_producer_mmf(); and example_kvs_producer_with_object_detection(); in the file /home/ubuntu/KVS_Ameba_Plumerai/ambpro2_sdk/component/example/kvs_producer_mmf/app_example.c

//example_kvs_producer_mmf();
//example_kvs_producer_with_object_detection();

cd ./KVS_Ameba_Plumerai 
cmake -DVIDEO_EXAMPLE=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../ambpro2_sdk/project/realtek_amebapro2_v0_example/GCC-RELEASE/toolchain.cmake -Sambpro2_sdk/project/realtek_amebapro2_v0_example/GCC-RELEASE -Bbuild 
cmake --build build --target flash_nn

scp -i kvs-keypair.pem ubuntu@54.64.xxx.xxx:/home/ubuntu/sample-kvs-edge_ai-video-streaming-solution/KVS_Ameba_Plumerai/build/flash_ntz.nn.bin ./flash_ntz.nn.bin

C:\Users\Administrator\Desktop\Pro2_PG_tool_v1.3.0>.\uartfwburn.exe -p COM3 -f flash_ntz.nn.bin -b 2000000 -U

./uartfwburn.arm.darwin -p /dev/cu.usbserial-110 -f ./flash_ntz.nn.bin -b 3000000

ATW0=myHotspotName 
ATW1=myPassword
ATWC