KKCompany saves 93% on data storage by migrating music streaming services to AWS

KKCompany Technologies (KKCompany) is an AI multimedia technology group based in Taipei, with offices across Asia. Its flagship music streaming platform, KKBOX, serves over 12 million users worldwide.

As its user base and music catalog rapidly grew, KKCompany faced significant challenges with its on-premises storage infrastructure. These included costly hardware scaling, constrained rack space, and ongoing maintenance demands that impacted engineering productivity. To address these limitations and support global expansion, KKCompany partnered with AWS and migrated its storage workloads to Amazon Simple Storage Service (Amazon S3), using Amazon S3 Intelligent-Tiering and on-the-fly encoding to optimize storage efficiency.

In this blog, we explore how KKCompany overcame its infrastructure bottlenecks and achieved a 93% reduction in storage costs, while ensuring a scalable, low-latency experience for millions of music streamers.

Challenges: operational overhead, scalability limits, and inefficient storage practices

KKCompany encountered multiple challenges managing its fast-growing media library on premises:

• Hardware scalability and rack space limitations: With media file growth averaging multiple terabytes each month, KKCompany’s on-premises data center quickly ran into physical limitations. Its rack capacity had reached maximum load, threatening the ability to store new content and expand into new markets.
• High operational overhead: Maintaining the on-premises infrastructure required engineers to spend 2–4 workdays each month in the physical server rooms to manage hardware. This significantly reduced engineering productivity and diverted focus away from business innovation and customer experience improvements.
• Inefficient file retention and format management: KKCompany historically stored every music file in multiple bitrates and formats, regardless of how often the file was accessed. Some tracks were only listened to once or twice but still occupied valuable storage space in multiple formats. This practice contributed to rapid storage expansion and made it difficult to identify candidates for tiering or archiving.
• Limited visibility into access patterns: Without fine-grained analytics, it was hard for KKCompany to determine which files could be safely archived without negatively impacting user experience. This uncertainty led to rarely accessed data in high-cost storage tiers by default.

These compounding issues created a storage environment that was expensive, inefficient, and difficult to scale with user demand.

Solution: Migrating to Amazon S3 and automating storage lifecycle management

To address these challenges, KKCompany migrated its entire media content repository to Amazon S3 with assistance from AWS Enterprise Support. This transition allowed them to eliminate the limitations of its physical infrastructure and fully embrace a scalable cloud-native storage model.

KKCompany adopted Amazon S3 Intelligent-Tiering, a storage class designed to automatically move data to the most cost-effective access tier as usage patterns evolve. This solution helped KKCompany eliminate manual lifecycle management while ensuring optimal storage cost savings without performance tradeoffs.

To further optimize storage costs, KKCompany combined S3 Intelligent-Tiering with on-the-fly encoding. On-the-fly encoding, also called just-in-time encoding, leverages AWS Lambda to encode files or media content during retrieval or delivery, based on demand. This approach reduces the need for pre-encoded files and minimizes storage usage, leading to further cost optimization and lower cloud storage expenses for customers. Historically, KKCompany stored multiple audio renditions (e.g., different bitrates) for each song, leading to excessive file duplication. Now, instead of storing each format, files are stored in its original, high-quality format and transcoded on demand at playback time. This dynamic transformation reduces redundant storage and streamline content management.

KKCompany designed robust multi-bucket architecture to balance cost optimization with user experience. Each music file is stored in two forms:

• A raw file (original format) placed in a raw bucket, eligible for transition to deep archive tiers.
• A base file encoded at 128kbps and stored in a base bucket for fast, low-latency access.

However, the standard retrieval times of opt-in archive tiers can range from 3–5 hours to within 12 hours. Even if the probability of accessing music files in archive tiers is very low, it could still happen. For example, when a sequel to a movie from the 1980s is released, users might want to listen to the soundtrack from the first movie for nostalgic reasons. Since the first movie was released over 30 years ago, the soundtrack may have already been moved to the Deep Archive Access tier of S3 Intelligent-Tiering. The standard retrieval time for the Deep Archive Access tier is within 12 hours. It is not practical for customers to wait hours for its music. To resolve this problem, KKBOX designed a base files bucket to ensure users can access music files without delays.

Let’s go through the detailed data flow with Figure 1:

Figure 1: Reference KKBOX searching and playing music flow

When a user plays a song, the process follows these steps:

1. User request initiation: When a user selects the desired quality and plays a track, the KKBOX client initiates a backend API call to handle the request.
2. Lambda-based file selection and processing:
   • 2.1: The backend API triggers a Lambda function that checks the availability of the raw file.
     • If the raw file is immediately accessible (in a hot storage tier), it is used to transcode the file to the desired bitrate.
     • If the raw file is archived, the system falls back to using the base file for immediate playback and simultaneously triggers a restore request for the raw file to move it to an accessible tier.
   • 2.2: The chosen file (raw or base) is processed to include digital rights management (DRM). The raw file is transcoded as needed, while the base file receives only DRM encoding. The processed file is then stored in the cache bucket.
   • 2.3: The API returns the CloudFront URLs for all available cached formats of the music file.
3. Playback: The KKBOX client then connects to the appropriate cached format hosted on CloudFront for uninterrupted playback.

By orchestrating this intelligent retrieval and processing pipeline, KKBOX ensures seamless access to music content while keeping long-tail storage costs minimized.

This approach allows KKCompany to maintain low-latency playback while capitalizing on S3 Intelligent-Tiering’s long-term cost benefits, even for rarely accessed files.

Benefits: 93% cost reduction and increased engineering agility

By adopting Amazon S3 Intelligent-Tiering and on-the-fly encoding, KKCompany achieved a 93% reduction in storage costs compared to its previous on-premises setup. Intelligent-Tiering’s automatic transitions eliminated the burden of manual data management, while on-the-fly encoding reduced the number of files that needed to be stored altogether.

This new architecture also freed up engineering resources, eliminating the need for monthly physical maintenance. With virtually unlimited scalability and a flexible cloud architecture, KKCompany is now well-positioned to scale KKBOX as listener demand grows across Asia and beyond.

Conclusion

KKCompany’s transition to AWS not only resolved pressing storage and scalability issues but also unlocked new efficiencies across its streaming platforms. By combining Amazon S3 Intelligent-Tiering with on-the-fly encoding, KKCompany reduced storage costs by 93%, eliminated operational overhead, and delivered a better experience for its global user base.

This success story highlights how cloud-native architectures can empower media companies to modernize legacy infrastructure and scale with confidence. Learn more about Amazon S3 Intelligent-Tiering and how it can optimize your storage strategy.