AWS Big Data Blog

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

Amazon OpenSearch Service now offers managed cross-Region disaster recovery for domains on OpenSearch 3.7 or later. It replicates data from a primary domain to a secondary domain in another Region with API-driven switchover and failover, providing an RPO under 1 minute and an RTO under 20 minutes.

Compaction keeps HBase read performance healthy, but when it runs during peak hours it drives latency spikes, write stalls, and RegionTooBusyException errors. In Part 2 of this series, you learn how to confirm compaction is the cause on Amazon EMR, tune the right parameters in the least-invasive order, and know when to stop and contact AWS Support.

Building a data foundation ready for agentic AI relies on a unified lakehouse and interoperability protocols like Model Context Protocol (MCP). This post walks through four pillars for building an agentic AI data foundation on AWS: a unified lakehouse, context engineering, multi-agent interoperability, and data governance.

Learn how to use the Amazon SageMaker Unified Studio Terraform module to provision a multi-team AI and analytics platform (domains, blueprints, project profiles, and projects) for data analytics, machine learning, and generative AI teams in a single terraform apply, with shared governance and least-privilege security defined entirely as code.

Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.

Amazon MSK bills at the cluster level, with no per-topic breakdown. This post shows how to allocate shared MSK cluster costs to individual Kafka topics by combining AWS Cost and Usage Report billing with per-topic CloudWatch or Prometheus metrics, giving each team a usage-based view of its spend.

Amazon SageMaker provides rich governance signals such as metadata forms, data quality scores, glossary coverage, and subscription data, but they live in separate APIs. This post shows how to join them into a single enriched Iceberg table and build a compliance scorecard you can query, score, and adapt to your governance policy.

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

How Confluent modernized its Amazon OpenSearch Service observability hot tier, ingesting hundreds of terabytes of logs per day, by migrating from r6gd to the purpose-built oi2 instance family. A measure-first approach cut the fleet 46%, roughly halved indexing latency, and cleared write backpressure, lowering cost even at a higher per-node price.