Senior Databricks Specialist / Data Architect

About the role

A hands-on senior Databricks specialist with proven experience in production enterprise environments is sought. The role focuses on independently designing, developing, troubleshooting, optimizing and governing Databricks solutions, beyond basic development. It covers enterprise-scale data solutions on Azure Databricks, including architecture, pipelines, governance and performance.

Key responsibilities

• Design and implement enterprise-scale data solutions on Azure Databricks, and define and maintain data architecture and platform standards.
• Build data ingestion and integration pipelines, and scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Implement Medallion Architecture (Bronze, Silver, Gold) and work with Delta Lake, optimizing Delta tables and pipelines.
• Design data models and transformations for analytical and operational reporting, and integrate Databricks with Microsoft Dynamics 365 / Dataverse and other applications.
• Develop batch and near-real-time pipelines, and implement data quality, validation, monitoring, lineage and governance.
• Optimize Spark jobs, clusters, SQL queries, Delta tables and pipelines for performance and cost; implement security and access controls.
• Work with Unity Catalog for governance, access control, cataloging and lineage, and support Power BI and other reporting platforms.
• Troubleshoot complex data engineering and performance issues, set development, testing and production standards, and support CI/CD and DevOps practices.
• Conduct technical reviews and mentor data engineers, ensuring solutions meet security, availability, scalability and disaster recovery requirements.

Required profile

• Minimum 8+ years of experience in data engineering or data architecture.
• Minimum 4+ years of strong hands-on experience with Azure Databricks.
• Strong experience designing enterprise data platforms and pipelines, and with large-volume enterprise datasets.
• Ability to troubleshoot complex production data issues.

Required skills

• Azure Databricks, Apache Spark and PySpark, Delta Lake, Medallion Architecture, Unity Catalog, SQL.
• Azure data services, Azure Data Factory and Azure Data Lake Storage Gen2.
• Data modeling, dimensional modeling, data quality and governance, performance and cost optimization.
• CI/CD, Git, Azure DevOps and deployment automation.
• Preferred: Databricks Certified Data Engineer Professional or Associate, Microsoft Certified: Azure Data Engineer Associate, or other relevant Databricks/Azure certifications.