Reduce MTTR with AI-driven RCA using AWS DevOps Agent and Splunk

Modern cloud-native applications generate rich telemetry across metrics, logs, and deployment histories. When performance degrades, operations teams have the data, but the challenge is correlating signals across multiple tools quickly enough to minimize customer impact. Root cause analysis remains a largely manual process dependent on institutional knowledge and operator experience.

This post shows how AWS DevOps Agent integrates with Splunk to automate end-to-end root cause analysis for distributed applications spanning AWS and external systems. We walk you through a sample healthcare claims processing application. A Splunk alert triggers an autonomous investigation in AWS DevOps Agent via webhooks. The agent queries AWS services and Splunk via the Splunk MCP Server and delivers a correlated root cause analysis with a mitigation plan. This reduces mean time to resolution (MTTR) from hours to minutes.

The MTTR challenge in distributed systems

Distributed applications span dozens of microservices, multiple AWS accounts, and hybrid infrastructure. Root causes can originate anywhere from a misconfigured connection pool, a deployment introducing an inefficient query, a downstream dependency hitting rate limits, to a subtle resource contention.

Traditional incident response follows a predictable, time-consuming pattern. An alert fires and an on-call engineer is paged. They manually query logs and metrics, form hypotheses, and cross-reference with deployment timelines. Eventually they identify probable cause. A senior engineer who built the service might resolve the issue in 30 minutes. A newer team member covering the same rotation might take hours or escalate.

The result is inconsistent MTTR, engineer fatigue, and a reactive operational posture that does not scale.

Solution overview

AWS DevOps Agent is an always-on operations teammate that autonomously investigates and proactively prevents incidents. Built on Amazon Bedrock AgentCore, it operates within Agent Spaces. These are isolated logical containers that bring together AWS accounts, telemetry sources, code repositories, CI/CD pipelines, and external tools into a single governed investigation context.

Within an Agent Space, the agent:

• Builds topology intelligence by auto-discovering resources and mapping dependencies across your application architecture.
• Investigates through structured hypotheses using specialized personas such as Triage, Root Cause Analysis, and Mitigation that systematically narrow to probable cause.
• Operates read-only via an AWS Identity and Access Management (IAM) role granting visibility across AWS services.
• Maintains an immutable audit journal, logging every reasoning step for compliance and review.
• Learns continuously by analyzing past investigations to build reusable investigative skills.
• Extends its reach via MCP, which can connect to external observability platforms like Splunk to query data beyond the AWS boundary.

The following figure shows how AWS DevOps Agent connects, with read-only access, to the AWS services it investigates and to external tools through MCP and webhooks.

Figure 1 – AWS DevOps Agent read-only connectivity to AWS services, with MCP and webhook integration to Splunk and other external tools.

Within the AWS account, the agent has read-only (“describe, get, and list”) access across Compute, CI/CD, Data and Networking, and Observability and Audit services. It connects to Splunk over the Model Context Protocol (MCP) and receives alerts through webhooks. For the complete list of supported integrations, including GitHub, GitLab, Jira, and PagerDuty, see Configuring integrations and knowledge.

Many operations teams rely on Splunk as their central observability platform with years of institutional knowledge encoded in saved searches, dashboards, and alert logic. AWS DevOps Agent meets teams where they are by integrating natively with Splunk through the Splunk MCP server, available on AWS Marketplace.

Prerequisites

Before following the walkthrough below, complete these prerequisites:

1. Create an Agent Space that includes your AWS accounts, repositories, and telemetry sources. Also review the AWS DevOps Agent CLI onboarding guide.
2. Connect the Splunk MCP server as a tool within your Agent Space. Authentication uses a bearer token created by your Splunk administrator.
3. Add the AWS DevOps Agent webhook URL to your Splunk webhook allow list. In Splunk Web, navigate to Settings → Server settings → Webhook allow list, and add the Agent Space webhook endpoint URL. This is required before Splunk can send alerts to AWS DevOps Agent.
4. Splunk uses bearer token authentication. Configure webhooks to receive alerts from Splunk and send webhooks from Splunk. In Splunk, select no authorization and instead use the custom header option.

For detailed configuration steps, refer to the AWS DevOps Agent documentation.

The integration provides three capabilities:

Direct Splunk access without pipeline changes
The agent connects to your existing Splunk deployment and autonomously constructs targeted queries during investigations. No changes to ingestion pipelines, index configurations, or saved searches are required.

Cross-domain correlation
The agent combines discoveries from Splunk, such as application logs, custom metrics, and business events, with AWS-native signals like Amazon CloudWatch metrics, AWS CloudTrail API activity, and deployment data from CI/CD pipelines. A spike in error logs from Splunk, correlated with an AWS Lambda configuration change, produces a root cause finding that neither tool could reach alone.

Investigation triggers from Splunk alerts
Investigations initiate directly from Splunk alerts via a webhook. Your existing alerting logic, rules refined over months, becomes the starting point for autonomous investigation rather than a page waiting for a human to respond.

Example: Healthcare claims processing incident

Consider a healthcare claims processing pipeline running in AWS as shown in Figure 2. Lambda functions orchestrate claim intake, Amazon Elastic Container Service (ECS) containers handle batch adjudication, Amazon Simple Queue Service (SQS) queues decouple processing stages, Amazon DynamoDB stores claim state, and Amazon Simple Storage Service (Amazon S3) holds document artifacts. A third-party clearinghouse API, outside the AWS boundary, handles claim submission and payer routing.

The following figure shows the healthcare claims processing architecture used in this walkthrough.

Figure 2 – Healthcare claims processing architecture showing AWS environment, observability bridge, and investigation layer with external clearinghouse API

The failure scenario: A sudden traffic spike hits the claims pipeline. Concurrent Lambda invocations surge, each making clearinghouse API calls. The external clearinghouse API exhausts its rate limit and begins returning HTTP 429 responses starting at ~17:25 UTC. Over 80% of claim submissions fail. The SQS queue starves, adjudication stops, and providers are blocked.

Step 1: Splunk detects the anomaly. A saved search triggers, when the claims success rate drops below threshold. It fires a webhook to AWS DevOps Agent with the error pattern and timestamp.

The following figure shows the Splunk dashboard where the anomaly first surfaces

Figure 3 – Splunk healthcare claims observability dashboard displaying success rate at 18:55 UTC

The following figures show the Splunk saved-search alert that dispatches a webhook to the AWS DevOps Agent when the success rate degrades.

Figure 4 – Splunk triggered alert showing claims success rate below threshold

Figure 5 – Splunk alert sent to DevOps Agent via webhook to trigger autonomous investigation

Step 2: Agent reads the learned topology. AWS DevOps Agent has auto-discovered the application’s resources and mapped their dependencies. The Triage persona uses this pre-built topology to immediately identify which components sit in the blast radius; no manual tracing is required.

Figure 6 – Auto-discovered application topology that the Triage persona traverses to identify affected components

Step 3: Agent receives the alert via webhook and tests hypotheses via Splunk MCP. The RCA persona tests multiple hypotheses in parallel:

• Internal infrastructure failure? The agent queries the CloudWatch metrics for the ECS adjudication cluster. CPU utilization, memory usage, and task count are all within normal operating range. It then checks for recent deployments or configuration changes that coincide with the incident timeline. None found. Internal infrastructure eliminated as a root cause.

Figure 7 – DevOps Agent receives webhook trigger and begins hypothesis-driven investigation

Figure 8 – DevOps Agent RCA persona queries Amazon CloudWatch and ECS container metrics confirming internal infrastructure is healthy

Figure 9 – Agent correlates Amazon CloudWatch metrics with deployment timeline to eliminate internal causes

• External dependency failure? Having ruled out internal causes, the agent constructs a targeted query via Splunk MCP to examine clearinghouse API response codes. Splunk returns HTTP 429, that is, “rate limit exceeded” responses beginning at ~17:25 UTC, confirming the clearinghouse rate limit was exhausted. This signal exists only in Splunk; clearinghouse API telemetry is not available in Amazon CloudWatch.

Figure 10 – Agent pivots to external dependency analysis using Splunk MCP after eliminating internal hypotheses

Figure 11 – DevOps Agent queries results via Splunk MCP, confirming clearinghouse API HTTP 429 rate-limit responses

• What triggered rate limiting? The agent correlates request volume over time and identifies a sudden surge in concurrent requests that exceeded the clearinghouse API rate limit within minutes. The evidence shows 2,737 total HTTP 429 responses versus only 655 successful (200) responses, representing an 80.7% failure rate sustained over the investigation window.

Figure 12 – DevOps Agent findings: timeline of events and key evidence showing rate-limiting at 17:25 and retry storm producing 80.7% failure rate

Step 4: Agent delivers mitigation plan. Root cause confirmed as external clearinghouse rate limiting due to traffic surge. The Mitigation persona produces the following recommendations.

• Immediate: Implement exponential backoff with jitter on clearinghouse calls.
• Short-term: Add circuit breaker pattern; queue claims for retry when rate limited.
• Prevention: Request rate-limit increase from the clearinghouse vendor.

Total elapsed time from alert to mitigation plan was under 8 minutes.

Figure 13 – AWS DevOps Agent Mitigation Plan

Clean Up

To avoid ongoing charges, remove resources in this order:

1. Remove the Splunk webhook alert. In Splunk, disable or delete the alert action that triggers AWS DevOps Agent (Splunk webhook alert actions).
2. Delete the Agent Space. In the AWS DevOps Agent console, select your Agent Space, choose Actions, then Delete agent space.
3. Delete IAM resources. In the IAM console, delete both ‘DevOpsAgentRole-AgentSpace-<suffix>’ and ‘DevOpsAgentRole-WebappAdmin-<suffix>’ roles, plus any customer managed policies prefixed with AIDevOps or any other identifier you used during onboarding.

For detailed instructions, see Deleting an Agent Space in the AWS DevOps Agent User Guide.

Conclusion

Without Splunk, the agent sees the Lambda timeouts and SQS queue growth but cannot determine why the external API is failing. With Splunk connected via MCP, it identifies the specific 429 pattern, correlates it with the request volume spike, and pinpoints the external root cause. Without this automation, an engineer might spend hours manually writing queries and cross-referencing.

This pattern extends beyond healthcare. Any workload with external dependencies (payment gateways, shipping APIs, identity providers, partner services) can benefit from this integration. If your incidents routinely require correlating AWS telemetry with external system logs, this combination eliminates the manual effort. AWS DevOps Agent surfaces symptoms inside AWS. Splunk reveals the cause in external systems. Connected via MCP, they deliver complete root cause analysis spanning the full application stack.

Next steps

To continue building on this integration and expand your operational automation:

1. Explore custom investigative skills for AWS DevOps Agent to codify your team’s runbooks and institutional knowledge into reusable agent capabilities.
2. Build custom agents with tailored system prompts, curated tool sets, and specialized skills to automate operational tasks specific to your infrastructure.
3. Set up additional MCP tool integrations with GitHub, GitLab, Jira, PagerDuty, etc. to expand the agent’s investigative reach across your toolchain.
4. Review the AWS DevOps Agent pricing to understand the pay-per-use billing model for agent investigation time.