Portkey AI

Portkey  has been used as an AI gateway and observability layer for LLM applications. The main use case is to route requests across multiple LLM providers through a common API, monitor latency, cost tracking, prompts, and responses, and add reliability features such as fallback, retries, caching, and guardrails. It is especially useful when moving from experimentation to production because it provides a single control layer for model routing, prompt management, usage tracking, and operational visibility.

Whenever an API is called through Portkey , it displays the total time consumption that has been taken. Observability  is provided there. When it fails to connect to an LLM model, it attempts to use a fallback mechanism, which is helpful.

The most valuable features are the unified AI gateway, observability, price calculations, routing controls, and guardrails. The universal API makes it easier to switch between and compare models without rewriting large parts of the application. For the organization, Portkey was providing two endpoints, one for the US and another for Europe, which was easily manageable. Some models were supported for the US, some not, some for the EU, and some for both. Switching between providers is very easy with Portkey. The fallback and retry capabilities are also useful because production LLM applications need resilience when a provider is slow, rate-limited, or temporarily unavailable. Cost and latency tracking are valuable because LLM usage can become difficult to manage across teams, but these can be easily managed through Portkey.

Observability  is the best feature because at the production grid application, the focus is usually on observability, such as how the end user is using it, what the latency is, how many errors occurred within time frames, which models were used, and how many tokens were consumed. These aspects are easy to manage in Portkey.

Portkey has significantly improved the organization by streamlining the AI processes being followed. Portkey improves the workflow for centralizing LLM operations instead of every application team building its own logging, routing, fallback, and cost tracking logic. Portkey provides a shared layer for these capabilities so it can be synced between all teams. It helps reduce engineering efforts when experimenting with different providers and models. It also improves visibility into production behavior because request volumes, latency, cost, errors, and model usage can be seen in one place. For teams building multiple GenAI applications, the biggest improvement is standardization. Portkey makes it easier to enforce common practices across observability, reliability, and governance.

The main area for improvement is onboarding and trial experience.

Portkey is very useful for production teams, but it may feel more like infrastructure than needed for small prototypes. More guided setup templates for common use cases such as RAG chatbot or internal assistant, evaluation pipelines, and multi-provider failover could make adoption easier. If the onboarding process could be easier, that would be beneficial, as there have been issues faced during onboarding.

Portkey has been used for the last two years as it is an integral part of Rosh.

Portkey is reliable approximately 99 percent of the time. Some endpoints will occasionally be down. Over a six-month period, endpoints went down once or twice, but most of the time it works very well.

Portkey is designed for scale production uses. It is useful when multiple teams and applications need a consistent gateway for LLM applications, observability, prompt management, and governance. These aspects are good in terms of scalability. The scalability value becomes more obvious as the number of models, providers, applications, and users grows. For a single prototype, it may feel optional, but for production GenAI platforms with multiple applications, it becomes much more valuable.

For enterprise production, support quality is very important. Support was reached out to once on Discord. There is a support group on Discord where the team is very responsive. They provide a newsletter with updates and features. The support experience has been good.

Previous solutions were being used, but they were not considered optimal. AWS Marketplace  is also being used for some LLM models, but AWS  does not support all the models. Portkey is a better solution because it has everything in one place.

The best aspect in terms of cost saving is using Portkey cache, which was built-in to the packages. This literally saved token costs because cache tokens cost less to process. This was the best aspect in terms of productivity. The setup was also smooth because only a single Portkey library is needed, and then any models can be used at once. Whether using AWS Marketplace  models, Google models, or OpenAI models, everything is available in one place. There is no need to worry about multiple platforms or multiple packages.

Time has been saved for multiple developers because of having a single package to manage multiple models and multiple providers. This was the main ease gained.

The pricing is great. The organization only pays for the model being used. For Portkey pricing, the organization is covering that cost. Roche is taking care of it. The pricing is very reasonable for an organization already running production LLM workloads that needs observability, routing, guardrails, or other features. Portkey pricing is separate from the underlying LLM provider cost. The model providers are still paid for token usage, and Portkey adds the gateway, observability, caching, guardrails, and management layers on top.

Nothing has been purchased from the AWS  marketplace. Roche is a partner for Portkey and the organization is an enterprise for Portkey, utilizing Portkey features.

Portkey should be recommended for teams that are moving GenAI applications into production and need more control over reliability, cost, observability, and governance. It is especially useful if multiple LLM providers are being used and if changes to models are expected over time. The overall rating given for Portkey is nine out of ten.

Hybrid Cloud

Amazon Web Services (AWS)

What do you like best about the product?

What I like best about Portkey is that it brings structure to what is otherwise a very chaotic part of building AI products. When you're working with multiple LLMs, APIs, and edge cases, things break silently—and debugging becomes painful. Portkey acts as a unified gateway that gives you visibility, control, and reliability out of the box.

The biggest win for me is observability + control. Having centralized logs, request tracking, cost insights, and performance metrics in one place makes a huge difference. Instead of guessing what went wrong, I can actually see how prompts behave, where latency spikes happen, and how much each request costs.

It also simplifies multi-model integration. Rather than managing different APIs and retry logic across providers, everything runs through a single layer with built-in fallbacks, routing, and caching. That alone removes a lot of engineering overhead and lets me focus more on building features instead of infra.

Another big plus is cost optimization. Features like caching, usage tracking, and model routing help avoid unnecessary LLM calls and keep spend predictable, which is critical when scaling.

What do you dislike about the product?

What I dislike is that the platform can feel a bit complex initially. There’s a learning curve, especially if you’re new to LLMOps, and some areas like advanced analytics and documentation could be more polished.

What problems is the product solving and how is that benefiting you?

Portkey solves the problem of managing and scaling LLM integrations reliably. When building AI products, handling multiple model providers, debugging failures, tracking costs, and maintaining performance quickly becomes messy and time-consuming.

The biggest issue it addresses is lack of observability. Without a proper layer, it’s hard to understand how prompts are performing, where latency is coming from, or why certain responses fail. Portkey gives structured logs, request tracing, and metrics, which makes debugging and optimization much faster.

It also solves fragmentation across LLM providers. Instead of writing custom logic for each API, retries, and fallbacks, Portkey provides a unified gateway. This reduces engineering overhead and makes it easier to switch or combine models without rewriting core logic.

Another major problem is cost unpredictability. With usage tracking, caching, and smarter routing, Portkey helps control and optimize LLM spend, which becomes critical as usage scales.

What do you like best about the product?

We started using Portkey when our AI usage began getting messy across multiple models and providers. What stood out immediately was how it brings everything into one place without adding complexity.

The biggest win for us has been visibility. You can actually see what’s happening with every request — cost, latency, failures — which makes debugging and optimization way faster. The observability layer alone is worth it, especially when you’re running multiple use cases in production.

What do you dislike about the product?

The product is evolving quickly, which is great, but it also means some advanced analytics and customization options are still catching up. The UI can improve in a few areas.

What problems is the product solving and how is that benefiting you?

Another thing I really like is how easy it is to switch between models or add fallbacks without rewriting core logic. The unified API and routing save a lot of engineering time, and it removes the usual vendor lock-in headache.

Prompt management and guardrails are also well thought out. It’s not just a gateway — it actually helps you run AI systems more reliably with things like caching, retries, and policy controls built in.

Setup is straightforward, and the documentation is clear enough that you can get something running quickly.

What do you like best about the product?

The standout feature is the dashboard and analytics, which make observability and monitoring straightforward. Implementation is simple thanks to the available APIs, and there is a wide selection of LLM models and providers. It integrates smoothly into daily monitoring and troubleshooting routines.

What do you dislike about the product?

The documentation falls short and often requires users to figure things out on their own. Additionally, price tracking does not work universally across all models, and for air-gapped setups, pricing updates must be done manually, which can lead to delays in getting accurate information.

What problems is the product solving and how is that benefiting you?

This tool has been instrumental for my DevOps teams, providing deep observability and effective cost controls. It also strengthens reliability through features like routing, retries, and fallbacks, which together make our delivery process faster, more cost-efficient, and predictable for our product teams. Since adopting it, we have experienced quicker release cycles, a reduction in incidents, and more consistent spending across our RAG, ASR, and OCR services running on Kubernetes.