Senior Databricks Specialist / Data Architect

About the role

The employer is looking for a Senior Databricks specialist with extensive experience designing, implementing and optimizing enterprise-scale data platforms on Microsoft Azure Databricks. The role covers the architecture, development, optimization, governance, integration and performance of the Databricks platform and its integration with Dynamics 365 CRM and other enterprise systems.

Key responsibilities

• Design and implement enterprise-scale data solutions and define Databricks architecture and platform standards
• Build data ingestion and integration pipelines, including scalable ETL/ELT pipelines and batch and near-real-time processing
• Design data models and transformation processes for analytical and operational reporting, supporting Power BI and similar platforms
• Integrate Databricks with Dynamics 365 / Dataverse and other applications, and work with Unity Catalog for governance, access control and lineage
• Implement data quality, validation, monitoring, lineage and governance; ensure security, availability, scalability and disaster recovery
• Optimize Spark jobs, clusters, SQL queries, Delta tables and pipelines for performance and cost
• Support CI/CD and DevOps practices, conduct technical reviews and mentor data engineers

Required profile

• Minimum 8+ years of experience in data engineering or data architecture
• Minimum 4+ years of hands-on experience with Azure Databricks
• Proven experience in production enterprise environments, able to design, develop, troubleshoot, optimize and govern Databricks solutions independently

Required skills

• Azure Databricks, Apache Spark and PySpark
• Delta Lake and Medallion Architecture (Bronze, Silver, Gold)
• Unity Catalog, SQL and Azure data services
• Azure Data Factory and Azure Data Lake Storage Gen2
• Data modeling, dimensional modeling, data quality and governance
• CI/CD, Git, Azure DevOps and deployment automation