Senior Data Engineer

In this role you will build the data foundations for a large-scale migration, designing repeatable pipelines that move diverse source data into a single platform. You will collaborate with the migration team, data lead, and platform partners to ensure accurate ingestion, transformation, and delivery. You’ll implement automated checks, monitor data processing, and maintain auditable data lineage with secure handling. This position offers the chance to shape data architectures within a mission-driven environment focused on responsible digitalisation.

• Hybrid working model with some flexible working
• 25 days’ holiday
• Private Medical Health care via Vitality
• Pension contribution
• Life Assurance
• Professional certifications supported

• Collaborate with the data lead, analysts, platform team and source organisations throughout the delivery
• Build repeatable processes to receive, unpack, validate and ingest data from multiple sources
• Transform source extracts into consistent structures for migration to a single platform
• Develop maintainable ingestion and transformation code using SQL, Python, PySpark and dbt
• Implement automated checks to ensure completeness and accuracy of migration
• Monitor, diagnose and recover failed or incomplete data processing
• Maintain auditable records of data sources, mappings, transformations and outcomes with secure data handling controls

• Hands-on experience building and operating data pipelines in a structured migration environment
• Experience with Azure, AWS or Google Cloud
• Experience with Spark, PySpark, Databricks or comparable distributed processing technologies
• Experience building modular, tested and documented transformations using dbt or equivalent
• Strong SQL and Python skills for transformations, validation and reconciliation queries
• Experience scheduling, monitoring and recovering pipelines using cloud-native or equivalent orchestration tools
• Experience with Delta Lake and transactional tables
• Experience using Terraform or equivalent infrastructure as code to provision cloud environments

• collaboration
• trust and support within teams
• continuous development and learning
• SQL
• Python
• PySpark