AWS Lead Data Engineer

HCLTech is a global technology company, home to more than 220,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending December 2024 totaled $13.8 billion.

Job Summary:

We are seeking a highly skilled and experienced AWS Lead Data Engineer, who will build, and lead the development of scalable data pipelines and platforms on AWS.

The ideal candidate will have deep expertise in PySpark, Glue, Athena, AWS LakeFormation, data modelling, DBT, Airflow, Docker and will be responsible for driving best practices in data engineering, governance, and DevOps.

Key Responsibilities:

• Lead the design and implementation of scalable, secure, and high-performance data pipelines using PySpark and AWS Glue.

• Architect and manage data lakes using AWS Lake Formation, ensuring proper access control and data governance.

• Develop and optimize data models (dimensional and normalized) to support analytics and reporting.

• Collaborate with analysts and business stakeholders to understand data requirements and deliver robust solutions.

• Implement and maintain CI/CD pipelines for data workflows using tools like AWS CodePipeline, Git, GitHub Actions.

• Ensure data quality, lineage, and observability.

• Mentor junior engineers and establish coding and design standards across the team.

• Monitor and optimize performance of data pipelines and storage solutions.

• Stay up to date with emerging technologies and recommend improvements to the data platform.

Required Skills & Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

• 10+ years of experience in data engineering.

• Strong hands-on experience with AWS services: S3, Glue, Lake Formation, Athena, Redshift, Lambda, IAM, CloudWatch.

• Proficiency in PySpark, Python, DBT, Airflow, Docker and SQL.

• Deep understanding of data modeling techniques and best practices.

• Experience with CI/CD tools and version control systems like Git.

• Familiarity with data governance, security, and compliance in cloud environments.

• Excellent communication and stakeholder management skills.

Preferred Qualifications:

• AWS Certified Data Analytics – Specialty or Solutions Architect certification.

• Experience with Terraform or CloudFormation for infrastructure as code.

• Knowledge of modern data architecture principles.