Sold by: TrueFoundryÂ
Unified AI Gateway: TrueFoundry's AI Gateway provides a standardized interface to access and manage over 250 LLMs, including both open-source and proprietary models, through a single API .
Overview
TrueFoundry's AI Gateway offers a comprehensive platform for managing large language models (LLMs) across diverse environments. With a unified API, it facilitates seamless integration with over 250 LLMs, including support for embedding, reranking, and real-time models. The platform ensures secure and centralized key management, allowing for the deployment of any Hugging Face model into the Gateway.
Observability is a cornerstone of the AI Gateway, providing real-time analytics to track usage, costs, and latency. Users can record all requests and responses, gaining full visibility into operations. Advanced filtering and custom metadata support enable deeper insights into model performance and user interactions.
To optimize performance and prevent overuse, the platform allows for precise rate limits at the team, user, and model levels. Role-Based Access Control (RBAC) ensures secure, permission-based access, while service accounts facilitate seamless authentication and automated workflows. Truefoundry
The AI Gateway is designed for high performance and reliability, featuring intelligent load balancing, failover mechanisms, and automatic retries to maintain seamless uptime. Its ultra-low latency capabilities process high requests per second (RPS) in just milliseconds, ensuring efficient real-time inference.
For prompt management, the platform offers centralized version control, allowing users to compare and test multiple prompts. Reusable prompt frameworks and integration with custom guardrails enable consistent and secure prompt deployment across applications.
Highlights
- Unified API Access to 250+ LLMs: Streamline integration with a wide array of LLMs through a single, standardized API, simplifying development and deployment processes.
- Comprehensive Observability & Insights: Gain real-time visibility into model usage, performance metrics, and operational costs, facilitating informed decision-making and optimization.
- Robust Access Control & Performance Optimization: Implement granular rate limits and RBAC to ensure secure access and optimal resource utilization, while intelligent load balancing and low-latency processing maintain high system reliability.
Details
Sold by
Delivery method
Deployed on AWS
Unlock automation with AI agent solutions
Fast-track AI initiatives with agents, tools, and solutions from AWS Partners.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on the duration and terms of your contract with the vendor. This entitles you to a specified quantity of use for the contract duration. If you choose not to renew or replace your contract before it ends, access to these entitlements will expire.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Description | Cost/month |
|---|---|---|
LLMGTW | LLM Gateway Contract | $1,000.00 |
Vendor refund policy
All fees are non-cancellable and non-refundable except as required by law.
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA)Â .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
Software as a Service (SaaS)
SaaS delivers cloud-based software applications directly to customers over the internet. You can access these applications through a subscription model. You will pay recurring monthly usage fees through your AWS bill, while AWS handles deployment and infrastructure management, ensuring scalability, reliability, and seamless integration with other AWS services.
Resources
Vendor resources
Support
Vendor support
For support related to our products, you can contact us via our support page at https://www.truefoundry.com/support or email us at support@truefoundry.com .
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
LiteLLM LLM Gateway (Proxy Server)
By LiteLLM
OpenAI Proxy Server (LLM Gateway) to call 100+ LLM APIs using the OpenAI format Bedrock, Huggingface, VertexAI, TogetherAI, Azure OpenAI, OpenAI, etc. Get started with Opensource LiteLLM here: https://github.com/BerriAI/litellm (12,000+ Github Stars)
TrueFoundry
By TrueFoundry
TrueFoundry offers an AI platform designed to streamline the training and deployment of machine learning models on Kubernetes. With a focus on cost efficiency and rapid deployment, the platform empowers teams to leverage ML technologies effectively.
Private AI Gateway on AWS based on LiteLLM
By Chaos Gears
An AI Gateway acts as the central control layer for all your LLM traffic - managing logging, retries, cost tracking, and robust routing. In this project, we deploy an open-source LiteLLM Server on AWS Fargate (with private ALB), configured for auto-scaling, metrics, and graceful shutdowns. We set it up as your unified API endpoint to models like Amazon Bedrock, OpenAI, Anthropic, and more. You get a fully functional gateway you can run in production, with deep visibility, centralized governance, and predictable billing. It’s perfect for teams that want reliable, consistent LLM access without engineering complexity.
Lasso for Secured Gateway for LLMs
By Lasso Security
Lasso Security provides comprehensive protection for Generative AI applications, addressing existing and emerging threats to ensure data integrity and privacy.
Customer reviews
0 ratings
0 AWS reviews
|
47 external reviews
Star ratings include only reviews from verified AWS customers. External reviews can also include a star rating, but star ratings from external reviews are not averaged in with the AWS customer star ratings.
Rajat S.
Great product for Building and Deploying highly scalable Gen AI Solutions with minimum overhead.
Reviewed on Sep 11, 2024
Review provided by G2
What do you like best about the product?
Ease of Use
Ease of Integration
Scalable deployments
Ease of Implementation
Customer Support
Number of Features
Ease of Integration
Scalable deployments
Ease of Implementation
Customer Support
Number of Features
What do you dislike about the product?
Not suitable for cost sensitive environment.
What problems is the product solving and how is that benefiting you?
Adoption of Gen AI for scalable and production ready Solutions
Suman P.
Truefoundry has streamlined the testing deployment and maintenance of models in the org
Reviewed on Sep 02, 2024
Review provided by G2
What do you like best about the product?
It has streamlined the deployment process. The steps that has personally helped me:
1. Ease of deployment from local/repo
2. Performance and fuctional testing of solution before prod deployment
3. Deployment metrics tracking and logging
4. Customisability in prod data storage which helps in maintenance and debug purpose
5. Tracking all the jobs and deployments in one place
Other than that their customer support is very prompt
1. Ease of deployment from local/repo
2. Performance and fuctional testing of solution before prod deployment
3. Deployment metrics tracking and logging
4. Customisability in prod data storage which helps in maintenance and debug purpose
5. Tracking all the jobs and deployments in one place
Other than that their customer support is very prompt
What do you dislike about the product?
If there is one scope that could be user friendly development environment which would make it a one stop platform for development testing and deployment of solutions
What problems is the product solving and how is that benefiting you?
Deployment and Maintenance of DS solutions
Computer Games
Faster way to deploy
Reviewed on Sep 02, 2024
Review provided by G2
What do you like best about the product?
Streamlined process of mode deployment and monitoring
What do you dislike about the product?
Nothing specific, process was pretty streamlined
What problems is the product solving and how is that benefiting you?
Ease of ML model deployment
Computer Games
Good place to integrate entire deployment pipeline
Reviewed on Sep 02, 2024
Review provided by G2
What do you like best about the product?
Very easy to use for any data scientist for ML model deployment
What do you dislike about the product?
Sometimes, few features like graphs and logs fails.
What problems is the product solving and how is that benefiting you?
Very easily any data scientist can deploy models without much configuration
Financial Services
Supercharges any developer workflow.
Reviewed on Aug 30, 2024
Review provided by G2
What do you like best about the product?
Prototyping/Testing deployments have become very quick with one click deployments, able to build and deploy through a particular branch and commit helps quick iterations. Ability to use SSH server with required service accounts and VsCode helps increase productivity. Support is helpful as well for any queries that occur. Able to look to live logs in the UI itself helps in quick debugging. We were able to deploy a hugging face model for a customer facing service fairly quickly using their ML model deployments.
What do you dislike about the product?
Some features such as RBAC control, safe guarding production deployments is needed. Approval based deployment rollout is also a good feature to have.
What problems is the product solving and how is that benefiting you?
Developer does not need to get his hands dirty with devops stuff. Plug and Play product where you just select what you want and don't need to understand much about the internals. Quick iterations are important, and deployments give just that.