Hasxxxxxx - Data Engineer Job Profile in Chicago, IL

• Over 9 years of professional experience in Information Technology expertise in Bigdata using Hadoop framework and Analysis, Design, Development, Documentation, Deployment, and Integration using SQL and Big Data technologies.

• Expertise working with AWS cloud services like EMR, S3, Redshift, EMR cloud watch for big data development.

• Good working knowledge of Amazon Web Services (AWS) Cloud Platform which includes services like EC2, S3, ELB, IAM, DynamoDB, Cloud Front, Cloud Watch, Route 53, Elastic Beanstalk (EBS), autoscaling, Security Groups, EC2 Container Service (ECS), Dynamo DB, Auto Scaling, Red shift, CloudWatch, CloudFormation, IAM, SQS, SNS, SES.

• Experience in building data pipelines using Azure Data factory, Azure data bricks and loading data to Azure data Lake, Azure SQL Database, Azure SQL Data warehouse and controlling and granting database access.

• Hands on experience working the Azure Cloud. (ADLS, ADF, Azure SQL Server, Synapse).

• Proficient in orchestration tools such as Apache Airflow and Oozie, and in automating infrastructure with Terraform across AWS, Azure, and GCP.

• Expertise in writing end to end Data processing Jobs to analyze data using MapReduce, Spark, and Hive.

• Worked on streaming data to consume data from KAFKA topics and load the data to landing area for reporting in real time and developed Kafka producers and Kafka Consumers for streaming millions of events per second on streaming data.

• Extensively used Terraform in AWS Virtual Private Cloud to automatically setup and modify settings by interfacing with control layer for multi cloud deployment using single configuration.

• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the usage patterns.

• Experience in importing and exporting data using Sqoop from HDFS to Relational Database systems and vice-versa and load into Hive tables, which are partitioned.

• Experienced in data modeling, data quality, designed and developed CDC, Pivot, Aggregator, Funnel, Merge, look up, Pivot, Transformer, Copy, Java Integration, XML stage, XML Output, calling java API’s, calling RESTful services and passing data to RESTful services jobs using IBM DataStage.

• Proficiency in SQL across several dialects (we commonly write MySQL, PostgreSQL, Redshift, SQL Server, and Oracle).

• Expertise in designing complex Mappings and have expertise in performance tuning and slowly changing Dimension Tables and Fact tables.

• experience in Analysis, Design, Development, Implementation, Testing and Support of Data Warehousing and Data Integration Solutions using Informatica PowerCenter.

• experience in using Informatica PowerCenter (7.1.3/8.6.1).

• Extensive testing ETL experience using Informatica 9.1/8.6.1/8.58.1/7.1/6.2/5.1 (Power Center/ Power Mart) (Designer, Workflow Manager, Workflow Monitor and Server Manager) Teradata and Business Objects.

• Extensively used Microservices and Postman for hitting Hadoop clusters. Installation, configuration, and administration experience in Big Data platforms Cloudera Manager of Cloudera, MCS of MapR.

• Migration of on-premises data (Oracle/ SQL Server/ DB2/ MongoDB) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).

• Good experience with use-case development, with Software methodologies like Agile and Waterfall.

• Ability to work effectively in cross-functional team environments, excellent communication, and interpersonal skills.