Data Engineer - Tech Lead (Databricks, Pyspark)

As a Senior Data Engineer – Tech Lead, you will design and optimize scalable, cloud-native data architectures with Azure Databricks, PySpark and Lakehouse principles. You lead data platform initiatives, guide modernization, and mentor engineers to deliver enterprise-grade data solutions. You will shape large-scale data ecosystems and enable AI-driven analytics while ensuring governance, reliability and cost efficiency. You will work closely with cross-functional partners to translate complex needs into robust technical strategies. This role offers impact across data strategy, implementation and team excellence.

• ESPP
• life, income, and critical illness cover
• private medical and dental
• Employee Assistance Program
• pension plan
• cycle/tech scheme and season ticket loans

• Lead architecture, design and build of large-scale data platforms using Azure Databricks
• Implement and optimize ETL and streaming pipelines with PySpark and Delta Live Tables
• Improve performance, control cloud costs and ensure reliability for streaming workloads
• Define governance, security and quality standards for the platform
• Collaborate with stakeholders to translate business requirements into technical solutions
• Develop integration approaches using Azure-native services (Data Factory, Synapse, Blob Storage)
• Mentor engineers, promote modern engineering practices and conduct technical reviews
• Drive CI/CD, Infrastructure as Code and automated testing in data engineering
• Implement observability and monitoring with Databricks Workflows and related tools
• Contribute to AI-driven initiatives by leveraging Databricks ML/MosaicML for generative AI and LLM solutions

• Bachelor’s or Master in Computer Science, Software Engineering or related field
• Extensive experience with Azure Databricks in production-grade platforms
• Expert PySpark with advanced optimization and data skew mitigation
• Strong Python programming and software design knowledge
• Experience with structured streaming, Delta Lake and Delta Live Tables
• Proven Lakehouse migration/modernization using Delta Lake or Apache Iceberg
• Proficiency with Azure cloud services and multi-cloud awareness (AWS or GCP)
• Hands-on CI/CD and Infrastructure as Code (Terraform, GitHub Actions, Jenkins)
• Strong leadership to guide teams and deliver in agile environments
• Excellent communication and stakeholder management

• leadership
• stakeholder management
• clear communication
• Azure Databricks
• PySpark
• Delta Lake