Lead Data Engineer

Lead Data Engineer - Government AWS Data Programme - SC eligible

About the role

Blackstone& is placing a Lead Data Engineer with a UK government department to set the technical direction for its data engineering work. The organisation runs an AWS data lake used by analysts and data scientists across the organisation. It needs someone to raise the standard of how data gets onto it and how it is looked after once it's there.

You'll lead a small team of data engineers and stay hands-on. You'll share the work, review code, unblock people and still write a fair amount yourself.

What you'll be doing

• Leading and line managing a team of data engineers, and growing their skills through pairing, review and coaching
• Designing how new source systems are integrated into the platform, from batch database extracts to API feeds
• Setting engineering standards for testing, CI/CD, documentation and data quality, and making sure they're followed
• Making architecture decisions across S3, Athena, Glue, Iceberg, dbt and Airflow, and writing them down so others can follow the reasoning
• Owning infrastructure as code in Terraform and keeping deployments repeatable
• Agreeing scope, priorities and delivery dates with product owners, system owners and senior data users
• Replacing manual or fragile pipelines with automated ones that can be monitored
• Representing data engineering in wider architecture and governance forums

What we're looking for

• A track record of leading data engineering teams and making technical decisions that held up
• Expert Python and SQL
• Deep AWS experience: S3, Athena, Glue, IAM and related data services
• Production experience with dbt and Airflow, or close equivalents
• Strong software engineering practice: pytest, git, GitHub Actions or similar CI/CD, and infrastructure as code (Terraform)
• Experience integrating complex operational systems into an analytics platform
• Confidence managing stakeholders up to senior level, including pushing back when needed
• Eligible for SC clearance

Nice to have:

• Apache Iceberg, the medallion (bronze/silver/gold) pattern, or Write-Audit-Publish
• Docker, Kubernetes or Spark
• Line management in a consultancy or client-facing setting
• Experience in government or another highly regulated sector, including data protection and audit requirements

Why this role

You'll have real influence over how a large public-sector data platform grows. Your team's work will reach decision-makers across the organisation.

ENGAGEMENT DETAILS

This engagement is offered outside IR35, reflecting the genuine independence and expertise the role requires. Key terms:

• Rate: Competitive
• Outside IR35 - engaged as an independent contractor via your own limited company or umbrella
• Initial contract length: 6 months, with extensions expected subject to performance and project progression
• Hybrid working - a mix of remote and on-site with the client as the engagement requires
• Start date: to be agreed. We are looking to move quickly for the right person