AWS Business Intelligence Blog

Integrate Amazon CloudWatch Logs with Amazon QuickSight using Amazon Athena

The ability to harness real-time metrics and insights is paramount for data analytics to help make informed business decisions. Amazon Web Services (AWS) provides a robust suite of tools designed to empower users with powerful data visualization and analysis capabilities. Amazon QuickSight, a serverless, cloud-based business intelligence (BI) service, powers data-driven organizations with unified BI at hyperscale. With QuickSight, users can meet varying analytic needs from the same source of truth through modern interactive dashboards, pixel-perfect reports, natural language queries, and embedded analytics.

In this post, we show how to seamlessly integrate Amazon CloudWatch logs with QuickSight using the Amazon Athena connector. With this integration, you can use your CloudWatch logs directly within the intuitive and feature-rich QuickSight environment, thereby unlocking deeper insights and streamlined workflows.

We demonstrate how to do the following:

  • Set up and store CloudWatch logs in a format that’s compatible with Athena
  • Set up the Athena connector in QuickSight, so you can seamlessly query your CloudWatch logs data as a federated data source
  • Create compelling visualizations and dashboards that provide actionable insights derived from your CloudWatch logs
  • Ask natural language questions against the CloudWatch data

This post can help equip data analysts, DevOps engineers, and BI professionals with the knowledge and tools needed to harness the full potential of your AWS ecosystem.

Solution overview

This post outlines how to visualize CloudWatch logs in QuickSight using Athena and the Athena CloudWatch connector.

Athena is an interactive analytics service that makes it simple to analyze data in Amazon Simple Storage Service (Amazon S3) using SQL. Athena is serverless, so there is no infrastructure to set up or manage, and you can start analyzing data immediately. You don’t even need to load your data into Athena; it works directly with data stored in Amazon S3.

CloudWatch is an AWS monitoring service for cloud resources and the applications that you run on AWS. You can use CloudWatch to collect and track metrics, collect and monitor log files, and set alarms. CloudWatch can monitor AWS resources, such as Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon DynamoDB tables, and Amazon Relational Database Service (Amazon RDS) DB instances, in addition to custom metrics generated by your applications and services, and log files that your applications generate, hosted on premises, hybrid, or on other clouds. You can use CloudWatch to gain system-wide visibility into resource utilization, application performance, and operational health. You can use these insights to react and keep your application running smoothly.

The following diagram illustrates the architecture for visualizing CloudWatch logs data in QuickSight.

The workflow consists of the following steps:

  1. The Athena CloudWatch connector is used to set up CloudWatch logs as a data source in Athena.
  2. Athena is used to set up views that can be ingested into QuickSight.
  3. Athena is set up as a data source in QuickSight, and then the metrics are visualized in QuickSight.

Prerequisites

For this walkthrough, you must have the following prerequisites:

Set up the Athena CloudWatch connector

Complete the following steps to set up the connector:

  1. On the Athena console, in the navigation pane, choose Data sources and catalogs.
  2. Choose Create data source.
  3. For Choose a data source, select Amazon CloudWatch Logs as the federated data source.
  4. Choose Next.
  5. On the Enter data source details page, for Data source name, enter the name that you want to use in your SQL statements when you query the data source from Athena.
  6. Under Connection details, choose Create Lambda function.
    This will open the Lambda function on a new tab.
  7. Review the information under Application settings and choose Deploy.
  8. In the previous browser tab, refresh the Lambda function choices in Connection details and choose the Lambda function you created.
  9. Optionally, for Tags, add key-value pairs to associate with this data source. For more information about tags, see Tag Athena resources.
  10. Choose Next.
  11. On the Review and create page, review the data source details, then choose Create data source.

The Data source details section of the page for your data source shows information about your new connector. You can now use the connector in your Athena queries.

Set up an Athena view

On the Athena console, in the navigation pane, choose Query Editor to access the query editor.

The following screenshot shows a view of the data sources available to query. CloudwatchLogs is the data source populated by the Athena CloudWatch connector.

For this post, we use an Amazon EC2 hosted sample web application to generate the access request logs for validating the connector. For instructions to set up the EC2 virtual machine with a sample Apache web application, see Install a web server on your EC2 instance. The database installation and setup sections are optional. This setup is for demonstration purposes; in practice, the same approach is applicable for other AWS compute resources.

The view definition based on the Athena table (CloudwatchLogs.default.access_log.<ec2-instance-id>) is created by the Athena CloudWatch connector. The log streams from the EC2 instances are included in the table all_log_streams.

The following query to create a new Athena view transforms the source table values to fetch key data elements such as the response code and formatted date/time for use in the QuickSight dashboard. You can adjust this view to change the format or the date range for fetching relevant records.

Create or replace view "AwsDataCatalog"."default"."cw_access-logs_insight" AS
SELECT log_stream, from_unixtime(time / 1000e0) as date_time, 
split_part( split_part(message, '"', 3) , ' ', 2) AS response_status, message
FROM "CloudwatchLogs"."access_log"."i-0fde7099dcb8162f7" where 
from_unixtime("time"/1000e0) >= DATE_ADD('day', -180, CURRENT_TIMESTAMP)

Set up QuickSight

Complete the following steps to set up QuickSight:

  1. Open the QuickSight console.
  2. On your profile menu, choose Manage QuickSight.
    You must be a QuickSight administrator to do this.
  3. Choose Security & permissions, then choose Add or remove.
  4. Select Amazon Athena, then choose Next.
    If it was already enabled, you might have to double-click it. Do this even if Athena is already enabled, so you can view the settings. No changes are saved until you choose Update at the end of this procedure.
  5. Enable the S3 buckets you want to access.
  6. To enable Athena federated queries, choose the Lambda function you created.
  7. To confirm your changes, choose Finish.
  8. To save changes to security and permissions, choose Update.
  9. On the QuickSight console, choose Datasets in the navigation pane, then choose New dataset.
  10. Choose the Athena card.
  11. Follow the prompts to create a new Athena data source using the resources you created.
  12. Choose Visualize to import the data into SPICE and start visualizing the CloudWatch logs.

Visualize your data

Complete the following steps to create a visualization:

  1. In the analysis sheet, add a new visual of type Vertical stacked bar chart.
  2. Drag and drop date_time into X Axis.
  3. Drag and drop message into Value.
  4. Drag and drop response_status into Group/Color.

The following screenshot shows our example visualization.

Use Amazon Q in QuickSight for Q&A on CloudWatch data

Using Amazon Q in QuickSight to perform natural language processing (NLP) on CloudWatch data empowers users to unlock actionable insights. By simply asking questions in natural language, you can instantly generate key insights, such as identifying anomalies in logs, pinpointing error trends, or uncovering operational metrics. Additionally, readers can perform “why” questions for contribution analysis, helping them understand underlying factors driving specific patterns. This capability extends the scope of CloudWatch data analysis, so you can seamlessly research logs. On top of QuickSight dashboards, you can use Amazon Q for real-time, intuitive interactions to improve operational efficiency and incident management.

To get started, create an Amazon Q topic using the CloudWatch Athena dataset in QuickSight. For more information, see Creating Amazon QuickSight Q topics.

You can now ask questions to the Amazon Q topic in QuickSight with NLP like “Show unique number of messages by date time.” Amazon Q can generate trend graphs with metrics by time dimension and also, with the new generative BI integration, provide additional text insights.

You can drill down further to a specific date or timestamp. From the same example, we can notice there is a spike in number of messages (January 11, 2025) and to gain deeper insights, you can ask follow-up questions to just focus on the day. Amazon Q can filter the date dynamically based on the prompt and show results for a specific date.

Conclusion

In this post, we showed how to capture and store CloudWatch logs, configure the Athena CloudWatch connector, create meaningful visualizations in QuickSight, and ask natural language questions using Amazon Q in QuickSight to drive informed decision-making. This solution demonstrated how to ingest data from CloudWatch logs in one AWS account for visualization in QuickSight. You can extend the same concept to ingest data from multiple AWS accounts. The ability to seamlessly integrate CloudWatch logs with QuickSight not only enhances your data analytics capabilities, but also streamlines your workflows, allowing you to focus more on insights and less on data wrangling. This integration can empower you to make data-driven decisions with confidence, such as identifying performance bottlenecks, monitoring application health, or simply gaining a deeper understanding of your operational metrics.

For additional discussion and help getting answers to your questions, check out the QuickSight Community.

About the authors

Vignesh Srinivasan is a Senior Solutions Architect at AWS. He previously worked with the Centers for Medicare & Medicaid Services (CMS), including helping to implement the Federal Health Exchange as part of the Affordable Care Act. He was also on the team that fixed healthcare.gov and successfully migrated the system to AWS. He has a master’s degree from the Rochester Institute of Technology and an MBA from the University of Maryland.

Channa Basavaraja is a Senior Solutions Architect at AWS with over 2 decades of experience building and scaling distributed business solutions for diverse industry segments. His areas of experience span AI/ML, data and analytics, event-driven architectures, and IoT and edge computing.

Ashok Dasineni is a Solutions Architect for Amazon QuickSight. Before joining AWS, Ashok worked with clients and organizations in the banking and financial domain, focusing on fraud research and prevention. He designed and implemented innovative solutions to improve business process, reduce cost, and increase revenue, enabling companies around the world to achieve their highest potential through data.