Ricoh Develops Its Own AI Model Using LLMs in Just 3 Months Using Amazon SageMaker

Having announced its goal to transform from an office automation manufacturer to a digital services provider in 2020, Ricoh launched its mid-term management strategy to increase digital services to account for 60 percent of sales by 2025. “We strive to develop and provide optimal solutions for our customers by using AI for image, voice, and natural language processing,” says Yoshiaki Umetsu, director of Ricoh’s Digital Technology Development Center, which helps set the path of the company’s digital services as part of the Digital Strategy division. Ricoh began developing AI in the 1990s and has worked on deep learning AI since 2015, applying it to visual inspections and vibration monitoring. In 2020, the company announced AI for Work, which uses a BERT-based natural language processing technology to analyze office documents and the voices of customers who call into its call centers for better work efficiency and customer assistance. However, because the BERT model has performance limitations caused by the necessity of supervised training, Ricoh then focused on large language models (LLMs) that can train on massive datasets, using an LLM-based open-source software model equivalent to GPT-3 to develop its own AI model.

“For a simple analogy, we enhanced a BERT model that has high school–level reading comprehension to a level where it can talk to people. That is our AI for Work,” says Umetsu. “Meanwhile, GPT-3 has the reading comprehension level of a new employee who graduated from a university so that it can write program code and proposal documents. We’re fine-tuning a GPT-3-comparable AI model with additional business data to raise its capabilities to the level of a practical worker.”

Ricoh aims to create a business-specific generative AI. For example, when used in multifunction printer development, it can understand that the term jam means “paper jam” rather than “confiture,” or that tray means “paper tray” and not “serving tray,” which generic generative AI cannot do. 

Ricoh adopted AWS to develop its own AI model equivalent to GPT-3. “We turned into a cloud solution to fulfill as many GPUs as required to develop an AI model within our budget and buy lead time,” says Umetsu. “However, GPU machines were hard to find anywhere in early 2022, and AWS was the only vendor that could provide many GPU instances immediately. Additionally, we weren’t familiar with large-scale, GPT-3 class AI engineering, so we asked for help from AWS.” This help was in the form of Amazon Machine Learning Solutions Lab, where experts help identify and build machine learning solutions.

Taking advantage of the AWS support program, Ricoh applied to relax the cap on Amazon Elastic Compute Cloud (Amazon EC2) P4d Instances and created a massive GPU environment. The company built a distributed learning environment using Amazon SageMaker—a service that is used to build, train, and deploy ML models for virtually any use case—as architecture to minimize learning time. The team initially stored training data in Amazon Simple Storage Service (Amazon S3)—an object storage service offering scalability, data availability, security, and performance—but during development, GPU processing speed outstripped data transfer rates, meaning training was sometimes performed with empty data or the data was lost during transfer. Ricoh solved the issue by transferring data to Amazon FSx for Lustre, which provides fully managed shared storage with the scalability and performance of the popular Lustre file system.

“We consulted experts from the ML Solutions Lab during development,” says Umetsu. “Our lack of experience and knowledge in AI development using massive amounts of GPU instances and huge storage capacities worried us. However, working alongside the Amazon Machine Learning Solutions Lab to develop the architecture, we accumulated our own AI engineering expertise and raised our skill levels so that we could solve problems in-house.”

In December 2022, Ricoh entirely developed its own AI model, Ricoh GPT, based on GPT-3, in three months. By training a GPT-3 equivalent model on a vast amount of Japanese language data, the company quickly built a model able to generate high-quality Japanese sentences.

“With companies announcing LLM development one after another, the fact that we were able to develop our own AI model in just three months and present it at a seminar is thanks in large part to using AWS,” says Umetsu.

The company is currently developing services using Ricoh GPT and considering providing vector search and custom GPT as search services. The vector search converts the proximity of words into numerical values to search based on the meaning of an entire document. When developers of multifunction devices want to know the cause of a malfunction and search a technology database in natural language, they receive highly similar results with evidence. However, vector searches have strong tendencies, and the expected answer may not be returned. In contrast, a custom GPT can derive more accurate answers by pretraining Ricoh GPT on company documents and technical data.

“For example, when an equipment maintenance worker simply speaks to custom GPT for a question through voice recognition, they will be presented with necessary technical information,” says Umetsu. “In the past, team members stationed in an office often answered questions from workers. Custom GPT will facilitate unattended operations.”

Ricoh is also working on “digital humans”: digital characters that converse with customers. Use cases include interactive signage using speech recognition, natural language processing, speech synthesis, and image generation, as well as automated customer service with AI avatars in the metaverse.

To prepare for an era in which digital assistance is everywhere, Ricoh will develop technologies, including reinforcement learning from human feedback and advanced AI, by expanding development resources for prompt engineering. The company is also considering launching an environment that spans from AI model development to the operation of next-generation GPT models (GPT.X), releasing software development kits dedicated to deep learning for GPT.X generative AI and using high-performance deep learning chips. For training chips, Ricoh is looking into AWS Trainium, which provides high performance for deep learning and generative AI training while lowering costs, and AWS Inferentia, which provides high performance at the lowest cost in Amazon EC2 for deep learning and generative AI inference.

Ricoh is also part of the AWS LLM Development Support Program for Japanese companies, through which AWS provides businesses with technical, business, and credit assistance for LLM development.

“Ricoh joined this program because we believe in AWS wanting to develop LLM in Japan,” says Umetsu. “We’ll continue to work on LLM research and development using AWS.”