This weblog put up is co-written with Govind Mohan and Kausik Dhar from Cognizant.
Migrating on-premises knowledge warehouses to the cloud is now not seen as an choice however a necessity for corporations to avoid wasting price and make the most of what the newest expertise has to supply. Though we’ve got seen loads of focus towards migrating knowledge from legacy knowledge warehouses to the cloud and a number of instruments to assist this initiative, knowledge is barely a part of the journey. Profitable migration of legacy extract, remodel, and cargo (ETL) processes that purchase, enrich, and remodel the information performs a key position within the success of any end-to-end knowledge warehouse migration to the cloud.
The normal method of manually rewriting a lot of ETL processes to cloud-native applied sciences like AWS Glue is time consuming and might be liable to human error. Cognizant Knowledge & Intelligence Toolkit (CDIT) – ETL Conversion Instrument automates this course of, bringing in additional predictability and accuracy, eliminating the chance related to handbook conversion, and offering sooner time to marketplace for clients.
Cognizant is an AWS Premier Tier Providers Companion with a number of AWS Competencies. With its industry-based, consultative method, Cognizant helps purchasers envision, construct, and run extra modern and environment friendly companies.
On this put up, we describe how Cognizant’s Knowledge & Intelligence Toolkit (CDIT)- ETL Conversion Instrument will help you robotically convert legacy ETL code to AWS Glue rapidly and successfully. We additionally describe the primary steps concerned, the supported options, and their advantages.
Answer overview
Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument automates conversion of ETL pipelines and orchestration code from legacy instruments to AWS Glue and AWS Step Features and eliminates the handbook processes concerned in a buyer’s ETL cloud migration journey.
It comes with an intuitive consumer interface (UI). You should utilize these accelerators by choosing the supply and goal ETL instrument for conversion after which importing an XML file of the ETL mapping to be transformed as enter.
The instrument additionally helps steady monitoring of the general progress, and alerting mechanisms are in place within the occasion of any failures, errors, or operational points.
Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument internally makes use of many native AWS providers, akin to Amazon Easy Storage Service (Amazon S3) and Amazon Relational Database Service (Amazon RDS) for storage and metadata administration; Amazon Elastic Compute Cloud (Amazon EC2) and AWS Lambda for processing; Amazon CloudWatch, AWS Key Administration Service (AWS KMS), and AWS IAM Id Heart (successor to AWS Single Signal-On) for monitoring and safety; and AWS CloudFormation for infrastructure administration. The next diagram illustrates this structure.
Methods to use CDIT: ETL Conversion Instrument for ETL migration.
Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument helps the next legacy ETL instruments as supply and helps producing corresponding AWS Glue ETL scripts in each Python and Scala:
- Informatica
- DataStage
- SSIS
- Talend
Let’s have a look at the migration steps in additional element.
Assess the legacy ETL course of
Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument allows you to assess in bulk the potential automation proportion and complexity of a set of ETL jobs and workflows which can be in scope for migration to AWS Glue. The evaluation choice helps you perceive what sort of saving might be achieved utilizing Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument, the complexity of the ETL mappings, and the extent of handbook conversion wanted, if any. You’ll be able to add a single ETL mapping or a folder containing a number of ETL mappings as enter for evaluation and generate an evaluation report, as proven within the following determine.
Convert the ETL code to AWS Glue
To transform legacy ETL code, you add the XML file of the ETL mapping as enter to the instrument. Person inputs are saved within the inner metadata repository of the instrument and Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument parses these XML enter information and breaks them all the way down to a patented canonical mannequin, which is then ahead engineered into the goal AWS Glue scripts in Python or Scala. The next screenshot reveals an instance of the Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument GUI and Output Console pane.
If any a part of the enter ETL job couldn’t be transformed fully to the equal AWS Glue script, it’s tagged between remark traces within the output in order that it may be manually fastened.
Convert the workflow to Step Features
The subsequent logical step after changing the legacy ETL jobs is to orchestrate the run of those jobs within the logical order. Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument permits you to automate the conversion of on-premises ETL workflows by changing them to corresponding Step Features workflows. The next determine illustrates a pattern enter Informatica workflow.
Workflow conversion follows the same sample as that of the ETL mapping. XML information for ETL workflows are uploaded as enter and Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument it generates the equal Step Features JSON file primarily based on the enter XML file knowledge.
Advantages of utilizing Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument
The next are the important thing advantages of utilizing Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument to automate legacy ETL conversion:
- Value discount – You’ll be able to cut back the general migration effort by as a lot as 80% by automating the conversion of ETL and workflows to AWS Glue and Step Features
- Higher planning and implementation – You’ll be able to assess the ETL scope and decide automation proportion, complexity, and unsupported patterns earlier than the beginning of the mission, leading to correct estimation and timelines
- Completeness – Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument presents one answer with assist for a number of legacy ETL instruments like Informatica, DataStage, Talend, and extra.
- Improved buyer expertise – You’ll be able to obtain migration objectives seamlessly with out errors brought on by handbook conversion and with excessive automation proportion
Case examine: Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument proposed implementation
A big US-based insurance coverage and annuities firm wished emigrate their legacy ETL course of in Informatica to AWS Glue as a part of their cloud migration technique.
As a part of this engagement, Cognizant helped the shopper efficiently migrate their Informatica primarily based knowledge acquisition and integration ETL jobs and workflows to AWS. A proof of idea (PoC) utilizing Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument was accomplished first to showcase and validate automation capabilities.
Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument was used to automate the conversion of over 300 Informatica mappings and workflows to equal AWS Glue jobs and Step Features workflows, respectively. Consequently, the shopper was in a position to migrate all legacy ETL code to AWS as deliberate and retire the legacy software.
The next are key highlights from this engagement:
- Migration of over 300 legacy Informatica ETL jobs to AWS Glue
- Automated conversion of over 6,000 transformations from legacy ETL to AWS Glue
- 85% automation achieved utilizing CDIT: ETL Conversion Instrument
- The client saved licensing charges and retired their legacy software as deliberate
Conclusion
On this put up, we mentioned how migrating legacy ETL processes to the cloud is important to the success of a cloud migration journey. Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument allows you to carry out an evaluation of the present ETL course of to derive complexity and automation proportion for higher estimation and planning. We additionally mentioned the ETL applied sciences supported by Cognizant Knowledge & Intelligence Toolkit (CDIT): ETL Conversion Instrument and the way ETL jobs might be transformed to corresponding AWS Glue scripts. Lastly, we demonstrated the right way to use present ETL workflows to robotically generate corresponding Step Features orchestration jobs.
To study extra, please attain out to Cognizant.
Concerning the Authors
Deepak Singh is a Senior Options Architect at Amazon Internet Providers with 20+ years of expertise in Knowledge & AIA. He enjoys working with AWS companions and clients on constructing scalable analytical options for his or her enterprise outcomes. When not at work, he loves spending time with household or exploring new applied sciences in analytics and AI house.
Piyush Patra is a Companion Options Architect at Amazon Internet Providers the place he helps companions with their Analytics journeys and is the worldwide lead for strategic Knowledge Property Modernization and Migration accomplice applications.
Govind Mohan is an Affiliate Director with Cognizant with over 18 yr of expertise in knowledge and analytics house, he has helped design and implement a number of large-scale knowledge migration, software raise & shift and legacy modernization tasks and works carefully with clients in accelerating the cloud modernization journey leveraging Cognizant Knowledge and Intelligence Toolkit (CDIT) platform.
Kausik Dhar is a expertise chief having greater than 23 years of IT expertise – primarily centered on Knowledge & Analytics, Knowledge Modernization, Software Growth, Supply Administration, and Answer Structure. He has performed a pivotal position in guiding purchasers via the designing and executing large-scale knowledge and course of migrations, along with spearheading profitable cloud implementations. Kausik possesses experience in formulating migration methods for complicated applications and adeptly setting up knowledge lake/Lakehouse structure using a big selection of instruments and applied sciences.