Senior Databricks Specialist / Data Architect

About the role

We are seeking a hands‑on Senior Databricks Specialist who can independently design, develop, troubleshoot, optimize, and govern enterprise‑scale data solutions on Azure Databricks. The role focuses on building robust, cost‑effective pipelines and data architectures that support analytical and operational reporting.

Key responsibilities

• Design and implement enterprise‑scale data solutions using Azure Databricks.
• Define and maintain Databricks architecture, standards, and Medallion (Bronze/Silver/Gold) design.
• Build scalable ETL/ELT pipelines with Apache Spark, PySpark, and Delta Lake.
• Integrate Databricks with Microsoft Dynamics 365 / Dataverse, APIs, and other enterprise applications.
• Implement data quality, validation, monitoring, lineage, and governance using Unity Catalog.
• Optimize Spark jobs, clusters, SQL queries, and Delta tables for performance and cost.
• Support Power BI and other reporting platforms with optimized data models.
• Establish development, testing, and production standards, and support CI/CD and DevOps practices.
• Mentor data engineers and conduct technical reviews.

Required profile

• Minimum 8 years of experience in data engineering or data architecture.
• At least 4 years of hands‑on experience with Azure Databricks.
• Proven experience designing enterprise data platforms and large‑volume pipelines.
• Strong understanding of data quality, governance, performance, and cost optimization.
• Experience with CI/CD, Git, Azure DevOps, and deployment automation.
• Relevant certifications such as Databricks Certified Data Engineer (preferred).

Required skills

• Azure Databricks
• Apache Spark & PySpark
• Delta Lake
• Medallion Architecture
• Unity Catalog (data governance, access control, lineage)
• SQL (large‑scale data processing)
• Azure Data Factory
• Azure Data Lake Storage Gen2
• Microsoft Dynamics 365 / Dataverse integration
• API integration with enterprise/CRM systems
• Data modeling & dimensional modeling
• Power BI
• CI/CD, Git, Azure DevOps