Overview
Migration of data environment and related functionality to AWS Glue and implement archiving policy for Raw files. â–Ş Check for possible optimization of ETL jobs during migration and align with client on the same. â–Ş Data Quality checks that are present in the current environment need to be implemented into the new AWS Glue jobs. â–Ş Glue jobs performance needs to beat or meet SLA of current workload. â–Ş Migrate Current data and historical data from Hadoop in Cloudera on prem environment to AWS Cloud. â–Ş Check any re-engineering required for Autosys jobs to ensure Orchestration integration is intact. â–Ş Configuration and Setup of business applications for scheduling â–Ş Ensure all the common services required for AWS Environment with respect to IAM, Alerts and Monitoring
Highlights
- Infrastructure Readiness • AWS Development Instance in Place ** • ETL and BI Instances in place ** • Connectivity between tools have been established ** ▪ Build out Development Target AWS environment ▪ Build out user security structure
- Target Data Store(S3) â–Ş Converted Data Store in Development environment â–Ş Converted access layer â–Ş Converted and new and Unit tested ETL code â–Ş ETL Scheduler changes complete for ETL code â–Ş Connectivity changes in place for dev and test servers ** â–Ş Historical Data Migration process set up and tested ** â–Ş Finalized End User security structure
- Instantiate new data model for Target Data Store â–Ş Build new pipes to feed into Target Data Store (S3) â–Ş Build and test History Data Migration Process â–Ş Connectivity Changes for downstream data feeds â–Ş Convert Business logic into transformation rules
Details
Unlock automation with AI agent solutions
