Arize AI

I'm using Arize AI for AI observability and evaluation for AI agents development.

I have it deployed in development and I need to check for specific questions what the agent was doing, what MCP tools it used, what thinking process it went through, what steps it chose to do in LangGraph, what nodes it went through, and if it had an error, what was the cause of the error. This way we can make the application better and better with iterations on the product. Arize AI, specifically the Phoenix tool, helps with developing this application.

Arize AI has an MCP server which is built-in. I can hook that up to Claude or GitHub Copilot and it's really helpful to extract sessions and conversations. I have key users who give me feedback about how the agent performed, and I mostly care about the negative feedback. I review with the MCP server and with the help of our coding agents what the agent deployed on development did. This way I can make the experience of the users better with each iteration. The tool also helps me with testing and evaluating the AI. I have a set of questions and I use LLM as a judge inside this tool to make at each release some assessment and tests with our agents. For each iteration I do these LLM as judge tests on a set of question datasets where I have YAML files and I have specified what the agent should do for those specific scenarios. I also have a golden dataset and I'm evaluating with that one as well using this tool.

I have improvements on the speed of development because with the help of Arize AI, I was able to cut on development speeds. Mostly on feedback from the client side and also from my point of view. When I as a developer deploy the agent, I have a set of questions which I verify and I can automate this process. A lot of testing and continuous improvement processes are automated using their tool.

Since using Arize AI, development speed improved by 50% to 60%. I saved approximately two days on each release on the testing and evaluation.

I think the most useful feature of Arize AI is their built-in MCP server which is providing a lot of data for our coding agents and our coding assistants in Claude or GitHub Copilot.

I can process data in bulk with Arize AI. I have approximately ten conversations with negative feedback and then it would take manually to review them two to five hours. With a coding agent and the MCP server hooked up, I can do the same review in approximately ten to twenty minutes. This helps me a lot with time management and efficiency. Also the AI agents and the coding assistant AI agents are able to come up with steps to fix those issues which I had. They receive a lot of data through that MCP server which would be impossible to give them manually. I think it's the most useful tool they have, and that's why I chose them.

I think Arize AI's UI could be better because it's pretty hard to understand. Before using their tool, I was using LangFuse and the UI from LangFuse was easier to comprehend. Also, their UI does not allow filtering for specific users or filtering on feedback. I think these minor UI changes would help a lot for users and also for non-technical users because business analysts are using the tool on my side. A lot of time they want to filter by a specific user and they cannot do that. I can do it with the MCP server, but filtering in the UI would be a great addition.

Another improvement would be some guidance on how to update your tool. If you are deploying the latest version of the tool right now, but in a month they have a new major release and then there are a lot of breaking changes, it's not easy to update it when it's deployed already. Descriptions or a guide for an AI agent or a guide for humans to deploy it would be really helpful.

I have been using Arize AI for six months.

I think Arize AI has quite good reliability. Some things could be improved, but based on my experience, as a developer I have to do some things so it works fine. I think the overall accuracy and reliability is quite good, and they are refining it with each release. I'm thankful and fulfilled with that.

Arize AI is very stable.

I think Arize AI is quite easy to scale. I did not have problems with it. It was straightforward. After deploying it, I could do replicas and everything quite easily. It's straightforward and easy.

I don't have experience with their customer support, but I have had a call with them. I think they are quite helpful and they provide useful insights.

I used LangFuse before.

I switched from LangFuse because it was a dependency hell. I had a lot of dependencies on LangFuse and a big percentage of them was not compatible anymore after some releases and after updates of some libraries. It was not compatible with my project. Also it could not be deployed because the deployment system was also not compatible with the new versions of LangFuse. I had to switch and I started with the team to look for a new tool and that's how I found Arize AI.

The return on investment is a lot of time saved. As I said before, I'm saving approximately two to three days per release, and it's a 50% to 60% reduction in time of testing after releases. It obviously has money implications because employees have to invest less time in the process of evaluation and observability. Also fewer employees are needed with the help of the MCP server because the coding agent can do what an employee should do manually. This way the employee has time to do other important things.

I don't know the costs because the company did this part. Setup was not straightforward, so costs were a little high time-wise. It took a lot of time to set it up.

I evaluated the most mainstream options. Mostly LangSmith was the other candidate.

I would rate Arize AI nine out of ten.

Their tool is really helpful and the things I mentioned about improvement are just minor things. It's not a big issue and would be nice to have, but overall their tools are really helpful for me.

They have quite good AI capabilities included in Arize AI. I appreciate the fact that I can choose any model I want and any LLM. That part is great. About security, I can't say much. I think they are doing a good job overall. It's straightforward to use any model I want. Overall, I think they are doing a good job.

I use private cloud for Arize AI.

As I know, I use AWS and also I use Azure from Microsoft.

I did not purchase Arize AI through the AWS Marketplace.

I had to see and read the setup steps or find some tutorial to set it up. If I didn't, it's not so easy to do it. I had to check on the database part. It has some environment variables which decide what database it uses. It took me a little time to find it. A good part of it is its built-in MCP server, which is a great addition to the tool.

Private Cloud

Amazon Web Services (AWS)

What do you like best about the product?

I really like how Arize AX makes it easier to understand what's happening with our AI. The tracing, monitoring, and evaluation tools are especially valuable to me. Tracing is particularly useful since it lets me look at individual AI interactions in more detail and understand what happened when a response didn't perform as expected. I also appreciate that I don't need to be extremely technical to get useful information from it, which is important for me.

What do you dislike about the product?

The learning curve. Arize AX has a lot of functionality, which is great, but it can sometimes feel a little overwhelming when you're first getting familiar with the platform, especially if you're not coming from a highly technical background. The onboarding could be improved by making it more guided and role-based. For example, someone using Arize AX from a product support perspective probably doesn't need the same level of technical details upfront as an engineer or data scientist.

What problems is the product solving and how is that benefiting you?

Arize AX helps me understand AI performance and identify issues without manual data digging. It saves time troubleshooting and provides insights to improve user experience.

What do you like best about the product?

What I like best about Arize Ax is the end-to-end visibility it provides into AI applications. You can trace an agent's workflow, quickly identify where something went wrong, and then use evaluations and experiments to understand whether a prompt, model, or workflow change actually improves performance. Having observability, evaluation, and experimentation together makes the feedback loop much faster than relying on logs or separate tools. In short, the best part is how easy it makes debugging AI systems. I can go from seeing that an agent is failing to tracing exactly what happened and evaluating the issue.

What do you dislike about the product?

The LLM/AI observability and evaluation platform has its drawbacks. My main criticism would be that if you only need basic LLM tracing, logging, and a handful of evaluations, Ax might be more infrastructure than you need. There are many concepts like traces, spans, datasets, experiments, evaluation, annotation, and Phoenix/AX terminology, which can take time to internalize. Once you move beyond standard evaluation into custom domain-specific evaluation, you may spend significant effort designing dataset prompts, scoring logic, and analysis workflows.

What problems is the product solving and how is that benefiting you?

Arize AX essentially solves the problem of making AI/LLM systems observable, debuggable, and reliable in production. When companies deploy AI agents and LLM applications, it can be difficult to answer traditional application monitoring, as it often isn't enough because an AI response can fail without the application itself technically crashing. Arize AX provides observability across the AI/agent workflow, including traces, model/LLM behavior, prompts, tool calls, retrieval, RAG steps, evaluation, latency, and costs, so teams can identify and fix problems faster.

What do you like best about the product?

The platform is also useful for teamwork because it gives developers, data scientists, and AI teams a shared place to understand how their applications are performing. Overall, I like Arize AX because it provides better visibility, makes AI problems easier to find, saves time during testing and debugging, and helps teams build AI applications that are more reliable and useful. I also like that it can help teams improve AI systems over time.

What do you dislike about the product?

The main thing I dislike about Arize Ax is that it can feel a bit difficult for new users when they first start using it. There are many features, settings, dashboards, and options, so it may take some time to understand everything properly. The platform is powerful, but sometimes the amount of information shown can feel overwhelming at first.

What problems is the product solving and how is that benefiting you?

Arize AX is helping solve many problems related to building, testing, and improving AI applications. One of the biggest challenges is that it can be difficult to understand why an AI model gives a wrong, poor, or unexpected answer. Arize AX makes it easier to test AI applications and see where issues are happening.

Arize AX also helps with debugging. When an AI response is not correct, it can be hard to pinpoint the exact reason. The platform provides better visibility into the AI process, which makes troubleshooting and improvement more straightforward.

What do you like best about the product?

What I like most about Arize AX is how helpful it is for monitoring AI workflows and spotting issues early. It gives me a clear view of how my models are performing, and it makes it easier to troubleshoot when I run into unexpected outputs.

What do you dislike about the product?

The biggest downside is that it can feel a little overwhelming at first, especially when you’re looking at a lot of metrics and traces. It also takes some time to set everything up and configure it in a way that’s genuinely useful for my specific workflow.

What problems is the product solving and how is that benefiting you?

Arize AX helps me understand what’s going on with AI models and workflows without having to manually dig through outputs. It makes it easier to catch unexpected results, troubleshoot problems, and track performance over time. That saves me time and gives me more confidence in the AI tools I rely on for my work.