Lead Data Engineer

As Lead Data Engineer, you will act as the technical authority for data engineering across a multi-cloud estate, focusing on scalable, enterprise-grade data platforms. You’ll split time between hands-on design and delivery and guiding engineers, stakeholders, and governance. Databricks and lakehouse concepts sit at the core of the platform, enabling analytics, data science, and AI use cases. You help shape data standards, architecture, and governance to deliver reliable, cost-aware solutions that support public services.

• Company car
• 5% on target bonus
• Private medical insurance
• Flexible working considered
• Pension – 6%
• Employee discounts and wellbeing benefits

• Define and enforce data engineering standards, patterns, and best practices across the organisation
• Architect and deliver scalable, resilient data solutions across Azure, AWS, and Databricks
• Lead complex engineering workstreams and ensure high-quality delivery outcomes
• Design and govern enterprise data models (lakehouse strategies, schema evolution)
• Champion and optimise Databricks-based platform (lakehouse, Unity Catalog, Delta Live Tables, streaming, MLflow)
• Drive a multi-cloud Databricks strategy enabling interoperable data platforms
• Optimize pipelines, compute, storage, and orchestration for cost-efficiency and scalability
• Define and implement CI/CD for data platforms with automated testing and secure deployments
• Embed data governance, security, and lifecycle controls (classification, encryption, access, lineage)
• Act as senior SME for troubleshooting, root-cause analysis, and long-term remediation
• Lead, mentor, and grow Data Engineers, promoting technical excellence
• Influence architecture and strategy across a matrix organisation and with senior stakeholders
• Communicate complex concepts clearly to technical and non-technical audiences

• Expert-level experience in enterprise-scale data engineering
• Advanced PySpark and SQL with performance optimisation expertise
• Strong data modelling and large-scale integration design
• Experience embedding testing, observability, and data quality controls
• Hands-on Databricks experience (lakehouse, Delta Lake, Unity Catalog, streaming)
• Broad cloud experience across Azure and AWS with multi-cloud capability
• Understanding of CI/CD for data platforms and environment promotion
• Experience in monitoring, observability, cost optimisation, governance, lineage at scale
• Excellent stakeholder management and leadership capabilities
• Strong communication skills to translate technical concepts into business value
• 8+ years’ data engineering experience in enterprise environments
• Relevant certifications (desirable): Azure Data Engineer Associate, AWS Data Analytics Specialty, or Databricks Data Engineer (Associate/Professional)
• SC clearance (or willingness to obtain it)

• stakeholder management
• leadership and mentoring
• clear communication
• PySpark
• SQL
• Databricks (lakehouse, Unity Catalog, Delta Lake, streaming)