CT Visa Reconciliation Tool

CT-VISA Recon is an automated data reconciliation and migration validation framework for Amazon Redshift. It compares a table in Amazon Redshift, provisioned or Serverless, with its counterpart in Databricks on AWS. It validates whether the two matches, from a pass/fail summary down to the individual cell. It replaces one-off SQL scripts and spreadsheet comparisons with a repeatable process. It suits Redshift-to-Databricks migrations, routine data quality checks, and ETL/ELT testing.

Teams connect Amazon Redshift and Databricks once, with each connection tested and saved. They then map source tables to target tables, define primary or composite keys, match columns, and add optional transformations. Bulk mapping can auto-match same-named tables across two schemas. Each reconciliation can add SQL filters on the Redshift and Databricks sides and run once, in bulk or on a schedule after every data load.

Recon offers eight validation types: schema and data type, row existence, duplicate key, row-level checksum, cell value, NULL value, distinct count and duplicate row. Fast mode gives a quick pass/fail health check on large Redshift tables. Detailed mode adds cell-by-cell comparison and shows exactly which records are missing, extra or changed. A dashboard, validation history and lineage view show reconciliation health across tables.

The application runs as a Databricks App in the customer's own workspace. The reconciliation job reads Redshift tables over JDBC and reads Databricks tables through Unity Catalog. Comparison jobs run on the customer's Databricks compute, and results and mismatch records are stored in the customer's own Unity Catalog tables. Credentials can be resolved from a Databricks secret scope, and data stays within the customer's AWS and Databricks environment.