Lead Data Engineer

Job Description

Lead Data Engineer (Global BI)

Location: - London

Employment Type: Full-Time

THE OPPORTUNITY

Are you a data engineer looking for your first lead role? If so, we want to speak to you!

We are seeking a Lead Data Engineer with deep expertise in Databricks and Amazon Web Services (AWS) to own the administration, architecture, and optimization of our cloud data platform. This role will serve as the technical leader responsible for designing, securing, and managing Databricks workspaces and AWS environments, while guiding data engineering projects from concept to deployment.

The ideal candidate is a hands-on technical expert who can collaborate across engineering, analytics, and business teams to deliver scalable data solutions, enforce best practices, and support the enterprise-wide data strategy.

YOUR PERSONALITY

We love different personalities at MiQ and strive to build teams constructed with diverse backgrounds and inclusive points of view. That said, we firmly believe that our values make us stronger and we look for people who share our values below.

• Passion – We love and live what we do.

Determination – We always find a way to ‘figure it out’.

Unity – We hold each other accountable. We win and lose as a team.

Agility – We anticipate the unexpected, embrace and adapt to change.

Courage – We dare to think unconventionally.

What You’ll Be Doing

As the Lead Data Engineer for the global BI team, you will be working closely with business stakeholders all around the world! You will oversee the Data Engineering capacity of a team split between India and the UK and work collaboratively with the wider data infrastructure teams.

Platform Architecture & Administration

 Design, implement, and maintain Databricks workspaces, clusters, job

configurations, Unity Catalog, and data governance frameworks.

 Administer and optimize AWS environments.

 Develop and maintain scalable, secure, and highly available data architectures.

 Monitor performance and cost usage of Databricks and AWS services; implement automation and optimizations.

Security, Governance & Compliance

 Define and enforce security, access control, and data governance policies across

Databricks and AWS.

 Implement best practices for encryption, monitoring, auditing, and compliance.

 Manage identity and permissions using IAM roles, Unity Catalog permissions, and workspace-level controls.

Project & Delivery Leadership

 Lead data engineering

projects from planning through deployment, ensuring

timelines, requirements, and deliverables are met.

 Collaborate with product owners, analysts, and data scientists to prioritize work and translate business needs into technical solutions.

 Mentor and guide data engineers, ensuring high-quality coding standards,

documentation, and operational excellence.

 Coordinate with cross-functional teams to integrate systems, pipelines, and tools.

Solution Engineering & Development

 Architect and build high-performance ETL/ELT pipelines using Databricks (Python, SQL, Spark).

 Manage ingestion from various sources.

 Implement DevOps and CI/CD practices for code deployment, environment

management, and version control.

 Support ML/AI workflows and advanced analytics workloads as needed.

YOUR QUALIFICATIONS And SKILLS

 Expertise administering Databricks (cluster management, jobs, notebooks, Unity

Catalog, security setup).

 Strong hands-on experience with AWS.

 Proficiency in Python, SQL, and PySpark.

 Strong understanding of distributed systems, data warehousing, and modern data lake architectures.

 Experience with CI/CD tools (Bitbucket, Jira etc).

 Excellent leadership, communication, and project management skills.

Who We Are

We’re MiQ, a programmatic media partner for marketers and agencies. We connect data from multiple sources to do interesting, exciting, business-problem-solving things for our clients. We’re experts in data science, analytics and programmatic trading, and we’re always ready to react and solve challenges quickly, to make sure our clients are always spending their media investments on the right things in the right places.

Our business keeps growing and our company keeps getting better because we keep hiring smart new people who are not afraid to challenge the “norm”. Infact, we encourage it!

If after reading all of this, you are as excited as we are, we’d love to hear from you.