Cloud Data Pipeline Developer (AWS/GCP/Azure) RM Jobs in New York, NY at Top Level Consultancy

We are looking for an experienced Cloud Data Pipeline Developer to design, build, and optimize scalable data pipelines on leading cloud platforms (AWS, GCP, or Azure). The ideal candidate will have strong expertise in cloud-native data services, ETL/ELT development, and hands-on experience with big data technologies.

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.

• Work with structured, semi-structured, and unstructured data sources.

• Implement data ingestion, transformation, and integration workflows.

• Leverage cloud-native services (AWS Glue, GCP Dataflow, Azure Data Factory, etc.).

• Optimize pipeline performance, scalability, and cost-effectiveness.

• Ensure data quality, reliability, and security across systems.

• Collaborate with data engineers, analysts, and business stakeholders to deliver data solutions.

• Implement monitoring, logging, and alerting for pipelines.

• Strong programming skills in Python, Java, or Scala.

• Expertise in at least one cloud platform (AWS, GCP, or Azure) and their data services.

• Hands-on experience with ETL/ELT tools and orchestration frameworks (Airflow, Prefect, Luigi).

• Experience with big data frameworks (Apache Spark, Hadoop, Kafka, Flink).

• Proficiency in SQL and working with relational/NoSQL databases.

• Knowledge of containerization and orchestration (Docker, Kubernetes).

• Familiarity with CI/CD pipelines and version control (Git).

• Experience with data warehousing solutions (Snowflake, BigQuery, Redshift, Synapse).

• Knowledge of streaming pipelines (Kafka, Kinesis, Pub/Sub).

• Exposure to machine learning data pipelines.

• Strong problem-solving and troubleshooting skills.

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.