Databricks Platform Lead - Schaumburg, IL (Hybrid)

Position : Databricks Platform Lead

Duration : 12 Months

Location : Schaumburg, IL (Hybrid)

Key Responsibilities :

Platform Leadership

Lead the design and deployment of large-scale data solutions on Databricks.

Establish best practices for notebook development, job orchestration, and cluster management.

Stay current with Databricks features and recommend improvements.

Data Engineering & Pipeline Development

Build robust batch and streaming data pipelines using PySpark, Delta Lake, and Structured Streaming.

Implement Medallion architecture (Bronze, Silver, Gold layers) for data curation.

Optimize Spark jobs for performance and scalability.

Cloud Integration

Integrate Databricks with AWS or Azure services (e.g., S3, EC2, Lambda, Glue, Azure Data Factory, Event Hub).

Ensure secure and compliant cloud architecture.

Data Governance

Implement data governance using Unity Catalog and Azure Purview.

Define access controls, lineage, and cataloging standards.

Cost Optimization

Monitor DBU consumption and cluster utilization.

Implement auto scaling, auto-termination, and right-sizing strategies.

Technical Leadership

Mentor junior engineers and conduct code reviews.

Lead technical discussions and decision-making.

Collaborate with data scientists, analysts, and business teams.

Required Skills & Qualifications

Bachelor's or Master's degree in Computer Science, Engineering, or related field.

10+ years in data engineering, with 3+ years in Databricks and Spark.

Expertise in PySpark, Delta Lake, Structured Streaming.

Experience with cloud platforms (AWS or Azure).

Familiarity with CI / CD, Git, and DevOps practices.

Strong understanding of data lakehouse architecture.

Preferred Qualifications

Databricks Certified Data Engineer.

Microsoft Certified : Azure Data Engineer Associate.

Experience with Kafka, Event Hub, and real-time data processing.

Knowledge of Dynamics 365, Power BI, and enterprise integration.

Soft Skills

Leadership and mentoring.

Strategic thinking and problem-solving.

Excellent communication and stakeholder management.

Adaptability and continuous learning.

Best Regards,

Vishal

vishal@smartcaliber.com

www.smartcaliber.com

Truth Lies in Heart