AWS Database Blog
Migrate a self-managed MySQL database to Amazon Aurora MySQL using AWS DMS homogeneous data migrations
Migrating your self-managed MySQL database to Amazon Aurora MySQL-Compatible Edition can significantly enhance your database’s performance, scalability, and manageability. AWS Database Migration Service (AWS DMS) makes this migration process straightforward and efficient. Homogeneous data migrations in AWS DMS simplify the migration of self-managed, on-premises databases to their Amazon Relational Database Service (Amazon RDS) equivalents. For the list of supported source databases, see Source data providers for DMS homogeneous data migrations. For the list of supported target databases, see Target data providers for DMS homogeneous data migrations.
In this post, we provide a comprehensive, step-by-step guide for migrating an on-premises self-managed encrypted MySQL database to Amazon Aurora MySQL using AWS DMS homogeneous data migrations over a private network. We show a complete end-to-end example of setting up and executing an AWS DMS homogeneous migration, consolidating all necessary configuration steps and best practices. By following this thorough walkthrough, you’ll gain practical insights into the entire migration process, from preparing your source and target environments to performing the final cutover.
Solution overview
Homogeneous data migrations are serverless and make it possible to migrate data between the same database engines, such as moving from a MySQL instance to an Aurora MySQL instance. With homogeneous data migrations, you can migrate data, table partitions, data types, and secondary objects such as functions, stored procedures, and so on. For more information, see Migrating data from MySQL databases with homogeneous data migrations in AWS DMS.
For homogeneous data migrations of the full load and change data capture (CDC) type, AWS DMS uses mydumper to read data from your source database and store it on the disk attached to the serverless environment. After AWS DMS reads your source data, it uses myloader in the target database to restore your data. After AWS DMS completes the full load, it uses native binary log replication to sync on-going changed to the target.
The following diagram shows the architectural overview to migrate a self-managed MySQL encrypted database to Aurora MySQL-Compatible using AWS DMS homogeneous data migrations.
You can use the same process to migrate an RDS for MySQL or self-managed MySQL database to Aurora MySQL-Compatible database hosted in a different VPC. To implement this solution, use the following steps:
- Prepare the source environment.
- Prepare the target environment.
- Create an AWS DMS subnet group.
- Import a certificate for in-transit encryption.
- Create secrets for the source and target databases in AWS Secrets Manager.
- Create an instance profile.
- Create data providers.
- Create a migration project.
- Create a data migration.
- Monitor replication.
- Perform cutover.
- Clean up.
Limitations
Keep in mind the following limitations:
- MySQL sources only support selection rules for the full load migrations. Selection rules allow you to choose the schema, tables, or both that you want to include in your replication. For more information, see Selection rules for homogeneous data migrations.
- You can’t use homogeneous data migrations in AWS DMS to migrate data from a higher database version to a lower database version.
- Homogeneous data migrations migrate encrypted MySQL databases and tables as unencrypted on the target database. This is because Aurora MySQL-Compatible and Amazon RDS for MySQL don’t support encryption using Keyring plugin. However, we recommend using AWS Key Management Service (AWS KMS) for encryption at rest for an Aurora MySQL DB cluster.
We recommend that you review Limitations for homogeneous data migrations before migrating your self-managed MySQL database to Aurora MySQL-Compatible.
Prerequisites
Make sure you meet the following prerequisites:
- You must have an active AWS account.
- The on-premises and the AWS account must have connectivity using either Direct Connect or AWS Site-to-Site VPN. For more information, see Network-to-Amazon VPC connectivity options.
- The on-premises security firewall should allow incoming traffic from Amazon Virtual Private Cloud (Amazon VPC) used by AWS DMS instance profile. During full load, AWS DMS connects to the on-premises database. For the ongoing replication, Aurora MySQL-Compatible connects to the on-premises MySQL database to enable binary log replication.
- You should have an AWS Identity and Access Management (IAM) policy and role for homogeneous data migrations.
- You need an Aurora MySQL database cluster in the target AWS account.
- IAM Role
dms-vpc-role
is available for AWS DMS to manage Amazon VPC. For more information, see Creating an IAM role for AWS DMS to manage Amazon VPC .
For the example in this post, we use the following configuration:
- AWS Region:
us-east-2
- Amazon VPC ID:
vpc-00000
- Target Aurora MySQL DB cluster:
aurora-mysql-01
- VPC security group attached to the Aurora MySQL DB cluster:
aurora-mysql-sg
- Binary log format:
ROW
- VPC security group attached to the AWS DMS instance profile:
dms-mtm-sg
- KMS key used to encrypt secrets in Secrets Manager:
"arn:aws:kms:us-east-2:<account-id>:key/mrk-d8e4262axxxx"
- Source database secret in Secrets Manager:
mysql-secret
- Target database secret in Secrets Manager:
aurora-secret
- On-premises source database host IP:
10.16.2.125/16
- IAM role for AWS DMS homogeneous data migrations:
HomogeneousDataMigrationRole-01
Prepare the source environment
In this section, you prepare the source database for homogeneous data migrations.
Create a database user
To run homogeneous data migrations, you must use a database user with the required privileges for replication. Use the following script to create a database user with the required permissions in the source MySQL database:
For more information on database user permissions, see Using a MySQL compatible database as a source for homogeneous data migrations in AWS DMS.
Enable binary logging and other required parameters
To configure CDC, you enable binary logging on the source database. To enable binary logging, you configure server-id
, log-bin
, binlog_format
, expire_logs_days
, binlog_checksum
, and binlog_row_image
parameters in the my.ini
(Windows) or my.cnf
(UNIX) file of your MySQL database. You must reboot the source DB instance for the changes to take effect. Refer Using a self-managed MySQL compatible database as a source for homogeneous data migrations for recommended values for these parameters.
Adjust binary log retention (binlog_expire_logs_seconds)
considering source database size and workload. If set too low and full load takes longer, binary logs may be deleted prematurely, compromising ongoing replication.
Network configuration
The on-premises security firewall should allow incoming and outgoing traffic between the on-premises network and the VPC CIDR range.
Prepare the target environment
In this section, we walk through the steps to prepare the target environment.
Create an IAM role for AWS DMS homogeneous migrations
To run homogeneous data migrations, you must create an IAM policy and an IAM role in your AWS account to interact with other AWS services. For more details on required IAM permissions, see Creating required IAM resources for homogeneous data migrations in AWS DMS. Complete the following steps:
- Create the
HomogeneousDataMigrationRole-01
IAM role: - Create the
AWSDMSHomogeneousRolePolicy-01
IAM policy and add the policy toHomogeneousDataMigrationRole-01
IAM role. This policy allows AWS DMS to perform the required steps for data migration. - Create the
AWSDMSHomogeneousRolePolicy-02
IAM policy and add the policy to theHomogeneousDataMigrationRole-01
IAM role. This policy allows AWS DMS to access the secrets in Secrets Manager for data migration.
Create a database user
AWS DMS requires a database user with certain permissions to migrate data into your target Aurora MySQL database. For more information on creating a database user and permissions, see Using a MySQL compatible database as a target for homogeneous data migrations in AWS DMS. In this post, you use the following script to create a database user with the required permissions in your MySQL target database:
For limitations on user name and password when using a MySQL compatible database as a target for homogeneous data migrations, see Limitations for using a MySQL compatible database as a target for homogeneous data migrations.
AWS DMS will also assign this user (dms_user
) as DEFINER of MySQL database objects. If you want to keep the DEFINER the same in Aurora MySQL-Compatible as your on-premises environment, you should consider one of following approach:
- Drop and recreate required objects such as triggers, procedures, functions, and views in Aurora MySQL after migration, ensuring objects have the required DEFINER.
- If you have
app_user
as DEFINER in the source database, you can createapp_user
in Aurora MySQL-Compatible and use this user to migrate to Aurora MySQL-Compatible.
Configure VPC security groups for the AWS DMS instance profile in the target AWS account
Complete the following steps to configure the aurora-mysql-sg
and dms-mtm-sg
VPC security groups:
- Create a VPC security group called
dms-mtm-sg
without any inbound rules in the target AWS account: - Verify the security group configuration using the following code:
- Fetch the security group ID of the
aurora-mysql-sg
security group, which is attached to your Aurora MySQL database: - Update the inbound rules of the security group
aurora-mysql-sg
by adding access fromdms-mtm-sg
:
Modify the DB cluster parameter group
Modify log_bin_trust_function_creators
to 1
in the custom DB cluster parameter group associated with your Aurora MySQL DB cluster. For more information, see Modifying parameters in a DB cluster parameter group. Changing this parameter allows AWS DMS to create functions and triggers in the Aurora MySQL database as part of the data migration.
Create a subnet group for AWS DMS migration project
A subnet group includes subnets from different Availability Zones which your instance profile can use. Note that a replication subnet group is a DMS resource and is distinct from subnet groups that Amazon VPC and Amazon RDS use. Create a subnet group for your AWS DMS instance profile. For more information, see Creating a subnet group for an AWS DMS migration project. In this post, you create a subnet group using private subnets on Amazon VPC:
Import the certificates for in-transit encryption
You can encrypt connections for source and target endpoints using SSL. To do so, you import a certificate (.pem file) to AWS DMS and assign that certificate to a data provider. Complete the following steps to import the certificate .pem file for the source MySQL database and Aurora MySQL DB cluster. You use these certificates at a later step to create data providers.
- Import the certificate for the source MySQL database:
- Refer to Download certificate bundles for Aurora to download the latest certificate bundle to import into AWS DMS.
- Import the certificate for the Aurora MySQL DB cluster:
Create secrets for the source and target databases in Secrets Manager
You store the database credentials information in the AWS Secrets Manager. AWS DMS uses this information to connect to your database. Complete the following steps to create the secrets:
- Create the secret for the source database user
dms_user
in Secrets Manager: - Create the secret for the target database user in Secrets Manager:
Create an instance profile
An instance profile specifies network and security settings for the serverless environment where your migration project runs. Create an AWS DMS instance profile to specify network and security settings for the database migration. You use this instance profile in a later step to create the migration project.
Create data providers
A data provider stores the information about your database. AWS DMS uses this information to connect to your database. Complete the following steps to create AWS DMS data providers for the source on-premises MySQL database and the target Aurora MySQL database. You use these data providers at a later step to create the migration project.
- Use the following configuration to create a data provider for the source on-premises MySQL database:
- Use the following configuration to create a data provider for the target Aurora MySQL DB cluster:
Create the migration project
Migration projects in AWS DMS are serverless. AWS DMS automatically provisions the cloud resources for your migration projects. You use a migration project to migrate data from your source database to a target database of the same type in the AWS. You specify the instance profile, source and target data providers, and secrets from AWS Secrets Manager while creating the migration project. Create the AWS DMS migration project with the data providers, secrets, and instance profile that you have created in the previous step:
Verify the migration project configuration. The project is now ready to use for homogeneous data migrations.
Create a data migration
After you create a migration project, you use the migration project for homogeneous data migrations. You create a new data migration in the migration project for the homogeneous data migrations. You can create several homogeneous data migrations of different types in a single migration project. Create a data migration in the migration project that you have created in the previous step:
After verifying your configuration, now you start the data migration:
You can check the data migration status as follows:
For more information on the checking the status and progress of the data migration, see Monitoring data migrations in AWS DMS.
Monitor replication
After you start your homogeneous data migration, you can monitor the replication status and CDC latency. To monitor how far the target DB is behind the source DB, connect to the target DB instance and run the SHOW REPLICA STATUS (Aurora MySQL version 3) or SHOW SLAVE STATUS (Aurora MySQL version 2). In the following command output, the Seconds_Behind_Source field tells you how far the target DB instance is behind the source:
To check the overall CDC latency in the units of seconds during the CDC phase, you can use the OverallCDClatency
Amazon CloudWatch metric. For more information on monitoring the data migration’s status and progress, see Monitoring data migrations in AWS DMS.
To view the metrics, use the following steps:
- On the AWS DMS console, in the navigation pane, choose Migration projects.
- Choose the project
mysql-migration-project-01
. - Navigate to the Data migrations tab.
- Choose Data migration, then
mysql-data-migration-01
. - Choose the Monitoring tab.
You can also use the following awscli commands to view the OverallCDCLatency
metrics:
Hint: You get DataMigrationExternalResourceId
from the Arn of the data migration created in earlier step – “arn:aws:dms:us-east-2:<account-id>:data-migration:PDhxxxrGgZHKpxoYGItvqfkmhyzWHmL”
You can also use the AuroraBinlogReplicaLag
CloudWatch metric associated with the writer instance to check the lag between the source and target database. For more information, see Instance-level metrics for Amazon Aurora.
Perform cutover
Once the replica lag is a near zero value, you are ready for cutover to point the application to the Aurora MySQL DB cluster in the target account. We recommend planning your cutover during a low traffic window and following your in-house business cutover checklist and processes. The following are key steps for the cutover process:
- Stop accepting connections on the source database.
- Make sure CDC latency from the source to the target DB instance is 0.
- To get the binary log position on the source, execute
SHOW MASTER STATUS
and compare the output with the target’s binary log coordinates, such asExec_Source_Log_Pos
andRead_Source_Log_Pos
and they should the same. - On the target Aurora MySQL instance, run
SHOW REPLICA STATUS
to obtain the binary log coordinates. - Check the
Seconds_Behind_Source
field to determine how far the target database instance is behind the source. - After stopping active database connections on the source database and checking the replication status, you stop the data migration
- Update the application configuration or DNS CNAME record with the target database endpoints.
- You can also set up replication from the target Aurora MySQL cluster to the source on-premises database using binary log replication to address fallback requirements before starting the application with the Aurora MySQL database.
- Start your application with the Aurora MySQL database.
Clean up
As part of this migration, you deployed several resources for AWS DMS, Secrets Manager, and other services in your AWS account. These resources will incur costs if they are in use. Be sure to remove any resources you no longer need. You can use following steps to delete the resources created in this post, make sure to replace the values based on your deployment:
Conclusion
In this post, we discussed the steps involved in migrating a self-managed MySQL database to Aurora MySQL-Compatible using homogenous data migrations in AWS DMS. We recommend testing the migration steps in non-production environments prior to making changes in production. We welcome your feedback. If you have any questions or suggestions, leave them in the comments section.