Principal Databricks Data Engineer
Astra-North Infoteck Inc.
12–18 years of overall Data Engineering experience
8+ years of experience with enterprise Data Warehouse and Data Lake platforms
5+ years of hands‑on experience with Databricks and Apache Spark at scale
Strong experience modernizing legacy Cloudera platforms including:
Modernize Cloudera platforms to Databricks Lakehouse architecture
Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
Ensure data reconciliation, audit integrity, and consistency during migration
Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
Curated / Conformed Layer
Semantic / Consumption Layer
Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
Strong experience with finance and risk data models including:
Financial Hierarchies
Liquidity Risk
Enable reporting capabilities including:
Build and manage semantic and consumption layers for BI, reporting, and analytics
Define business metrics, dimensions, hierarchies, and KPIs
Experience with:
Develop and optimize large‑scale data pipelines using:
Optimize workloads using:
Data Quality Frameworks
Establish data lineage and metadata management
Ensure data security, access control, and compliance standards
Experience with AWS or Azure cloud platforms
Experience with CI/CD pipelines using:
Familiarity with:
Experience with dbt is an advantage
Act as a technical authority and lead enterprise architecture decisions
Mentor senior engineers and establish engineering standards
Collaborate with finance, risk, analytics, and governance stakeholders
Translate complex data structures into business‑ready insights
Nice‑to‑Have Skills
Experience in BFSI, Capital Markets, or Regulatory Reporting
Oracle Financials
Experience supporting AI/ML workloads
Databricks and Cloud Certifications
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives
Shape enterprise finance and risk data platforms
Support regulatory, management, and analytical reporting systems
#J-18808-Ljbffr
8+ years of experience with enterprise Data Warehouse and Data Lake platforms
5+ years of hands‑on experience with Databricks and Apache Spark at scale
Strong experience modernizing legacy Cloudera platforms including:
Modernize Cloudera platforms to Databricks Lakehouse architecture
Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
Ensure data reconciliation, audit integrity, and consistency during migration
Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
Curated / Conformed Layer
Semantic / Consumption Layer
Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
Strong experience with finance and risk data models including:
Financial Hierarchies
Liquidity Risk
Enable reporting capabilities including:
Build and manage semantic and consumption layers for BI, reporting, and analytics
Define business metrics, dimensions, hierarchies, and KPIs
Experience with:
Develop and optimize large‑scale data pipelines using:
Optimize workloads using:
Data Quality Frameworks
Establish data lineage and metadata management
Ensure data security, access control, and compliance standards
Experience with AWS or Azure cloud platforms
Experience with CI/CD pipelines using:
Familiarity with:
Experience with dbt is an advantage
Act as a technical authority and lead enterprise architecture decisions
Mentor senior engineers and establish engineering standards
Collaborate with finance, risk, analytics, and governance stakeholders
Translate complex data structures into business‑ready insights
Nice‑to‑Have Skills
Experience in BFSI, Capital Markets, or Regulatory Reporting
Oracle Financials
Experience supporting AI/ML workloads
Databricks and Cloud Certifications
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives
Shape enterprise finance and risk data platforms
Support regulatory, management, and analytical reporting systems
#J-18808-Ljbffr
Reference: WJ-3875_16353614 · Posted 17 Nov 2026, 01:25 AM