Principal Databricks Data Engineer
Astra-North Infoteck Inc.
- 12–18 years of overall Data Engineering experience
- 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
- 5+ years of hands‑on experience with Databricks and Apache Spark at scale
- Strong experience modernizing legacy Cloudera platforms including:
- Modernize Cloudera platforms to Databricks Lakehouse architecture
- Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
- Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
- Ensure data reconciliation, audit integrity, and consistency during migration
- Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
- Curated / Conformed Layer
- Semantic / Consumption Layer
- Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
- Strong experience with finance and risk data models including:
- Financial Hierarchies
- Liquidity Risk
- Enable reporting capabilities including:
- Build and manage semantic and consumption layers for BI, reporting, and analytics
- Define business metrics, dimensions, hierarchies, and KPIs
- Experience with:
- Develop and optimize large‑scale data pipelines using:
- Optimize workloads using:
- Data Quality Frameworks
- Establish data lineage and metadata management
- Ensure data security, access control, and compliance standards
- Experience with AWS or Azure cloud platforms
- Experience with CI/CD pipelines using:
- Familiarity with:
- Experience with dbt is an advantage
- Act as a technical authority and lead enterprise architecture decisions
- Mentor senior engineers and establish engineering standards
- Collaborate with finance, risk, analytics, and governance stakeholders
- Translate complex data structures into business‑ready insights
Nice‑to‑Have Skills
- Experience in BFSI, Capital Markets, or Regulatory Reporting
- Oracle Financials
- Experience supporting AI/ML workloads
- Databricks and Cloud Certifications
Key Responsibilities
- Lead Cloudera to Databricks transformation initiatives
- Shape enterprise finance and risk data platforms
- Support regulatory, management, and analytical reporting systems
Reference: WJ-4483_1644133 · Posted 16 Nov 2026, 05:12 PM