IT & Software

Lead Data Engineer – Spark / Scala - Vice President

Citigroup

Belfast · County Antrim · United Kingdom

Overview

In this role you will engineer scalable data reconciliation pipelines on Databricks, migrating from Hadoop to cloud-native architectures on AWS. You will work at the intersection of data engineering and financial data operations, shaping a production-grade platform that supports downstream reporting, risk, and decision-making. The role focuses on modernising large-scale pipelines, improving data quality, and delivering reusable components across the data platform. You join a collaborative team that values practical engineering, architectural input, and continuous improvement, with mentoring and flexible work options.

Pay / Benefits
  • 27 days annual leave (plus bank holidays)
  • discretional annual bonus
  • Private Medical Care & Life Insurance
  • Employee Assistance Program
  • Pension Plan
  • Paid Parental Leave
Responsibilities
  • Build and optimise large-scale Scala Spark data pipelines on Databricks for reconciliation
  • Refactor legacy Hadoop pipelines to cloud-native Databricks/AWS architectures with Delta Lake and Workflows
  • Tune Spark jobs to remove bottlenecks and standardise patterns
  • Translate designs into detailed implementations, data models, and reusable components
  • Collaborate with architects, platform, and DevOps teams on design and production releases
  • Write and maintain tests (unit, integration, data validation) for quality and reliability
  • Contribute to engineering standards via shared frameworks and libraries
Key requirements
  • Experience with Scala Spark or PySpark or Java Spark
  • Familiarity with AWS services (S3, Glue, IAM, CloudWatch, Step Functions)
  • Exposure to data reconciliation or financial data domains
  • Experience with CI/CD pipelines and DevOps practices
  • Knowledge of data modelling concepts and orchestration patterns (beyond Databricks Workflows)
  • Experience with large-scale Hadoop to cloud migration or legacy platform modernization
  • Familiarity with streaming data processing (Kafka, Spark Structured Streaming)
  • Experience contributing to shared engineering frameworks or reusable libraries in a platform team
  • collaboration
  • problem solving
  • attention to detail
  • Scala Spark
  • PySpark
  • Java Spark

Reference: WJ-747_31291349 · Posted 13 Nov 2026, 04:54 AM

Find more on GigBlow

This role is listed on GigBlow for discovery and search. Hiring decisions and applications are handled by the employer or their chosen application system.

Data Engineer jobs in other states

Browse jobs