Indus : GenAI powered LLM

Tech Mahindra’s Indus GenAI solution empowers all Indic languages that have originated out of the great Indus Civilization. Tech Mahindra, with use of AWS powerful platform and services, have sought to build an Open-Source Large Language AI model to serve the needs of 25% of the world's population.

The Indus Project uses AWS S3 and EC2 instances and intends to make India based Foundational Model in Indic languages. In the current version Indus LLM supports Hindi and its 37 dialects.

AWS-Powered LLM model

The solution's robust architecture is deployed on AWS cloud platform and uses S3 and EC2 services for allowing full flexibility to run the solution.

1. High performance GPU based instance: The model uses AWS EC2 G5 instances for graphic intensive application and ML inference.
2. User Management and Content Delivery: Utilizes Amazon Cognito for secure authentication
3. Storage Powerhouse: Application also uses Amazon S3 for scalable content storage
4. Database Services: Solution uses RDS which makes it easier to setup, operate and scale relational database.

Key Features:

1. Built Indus LLM from scratch which is 1.2 Bn parameter model in Hindi and its 37 dialects
2. Built from 22 Bn tokens using cleaned Hindi and Dialects data.
3. Most economical, sustainable and ethical model
4. No spurious token has only Hindi and its dialects.
5. Features above Llama 2.0 model on Indic LLM Leaderboard for abstract reasoning.
6. Indus was deployed on AWS Sagemaker and tested for Inference to a decent performance.
7. NLP Toolkit- This toolkit is used for cleaning of any data other than Hindi and also transliterates dates and English text. This also removes bad words
8. Bias Removal Toolkit- We have annotated data created for bias and non bias. A cosine difference mechanism is applied as a classifier to find and remove biased sentences
9. Bias Removal Dashboard- Dashboard to represent biases removed

Benefits:

1. Helps to bridge digital gap for Hindi speaking masses of rural India Enables Indic Digital Transformation
2. Can be finetuned for specific use cases to cater to the needs of native Hindi users
3. Enables Digital transformation in Indic languages