Data Engineer Jobs in Texas City, TX at Technovera

Data Engineer 

Role Overview:
The Data Engineer II will be responsible for building and maintaining scalable data pipelines, managing delta lake architecture, and supporting analytics initiatives across the organization. The role involves strong knowledge in Azure Databricks, data engineering, and pipeline development. The candidate will collaborate closely with platform architects and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture, while also maintaining coding standards, migration processes, and CI/CD practices.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks, Apache Spark, and Python.

• Collaborate with data engineering teams to integrate data from various on-premise source systems into Azure.

• Ensure data quality, integrity, and security throughout the data pipeline.

• Automate unit test cases as part of data pipeline development.

• Contribute to the development and maintenance of Lakehouse architecture using Azure Databricks and Delta Lake.

• Ensure data governance and security within the Lakehouse environment.

• Execute proof of concepts (POCs) to evaluate new tools and concepts introduced by Azure/Databricks.

• Troubleshoot pipeline and Lakehouse architecture issues, resolving bottlenecks and errors.

• Optimize pipeline and Lakehouse performance, scalability, cost, and reliability.

• Collaborate with data science teams to integrate pipelines with data science workflows.

• Ensure seamless data exchange between engineering and science teams.

• Participate in peer reviews for coding standards.

• Serve as subject matter expert for PySpark and Scala.

• Perform additional duties as assigned.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.

• Certifications such as AWS Certified Solutions Architect, Cisco CCNP, Microsoft Azure Solutions Architect, or ITIL are preferred.

• Databricks and Azure certifications are a plus.

• Knowledge in cloud development practices.

• Strong expertise in cloud platforms (AWS, Azure, GCP), virtualization, and networking.

• Preferred knowledge with infrastructure automation tools (Terraform, Ansible, Kubernetes).

• Proficiency in Linux and Windows server administration.

• Solid knowledge of cybersecurity best practices and disaster recovery planning.

• Strong leadership knowledge in managing technical teams and project management.

• Knowledge of log management and monitoring tools (Splunk, Datadog, Prometheus).

• Knowledge of Talend, Power BI, and Qlik.

• Familiarity with containerization and microservices.