Job Description
Overview of Job Responsibility :
Develop and support data products, this resource plays a critical role in the development and maintenance of data solutions and deliverables used by internal and external stakeholders.
Design and develop big data solutions built on the Databricks and Azure technology stacks. You will help drive PHM data strategy, managing data pipelines, and ensuring the reliability and scalability of our data platforms.
You will report to the Director, Data Engineering.
Responsibilities :
- Design, implement, and maintain scalable data pipelines and architectures using Databricks and Azure technologies
- Create high performance data products by monetizing warehouse and ETL solutions using python, sql and spark
- Build data flow automation to retrieve, transform and load data. Manage data warehouse structure and file storage rules
- Performance tuning and optimization of ETL and database structures. Optimize data infrastructure performance, reliability, and cost-efficiency
- Monitor long running transactions and can improve query executions using best practices and methodologies
- Design data models for enterprise-wide data integration incorporating :
- Structured and unstructured data
- Real-time, on-demand, batch and varying timing schedules
- Collaborate with cross-functional teams to understand data requirements and deliver innovative solutions
- Develop and enforce data engineering best practices, standards, and policies
- Maintain best practices for handling, storing and using data
- Evaluate and implement new technologies and tools to support data engineering initiatives
- Drive continuous improvement and innovation in data engineering processes
Qualifications
Minimum Requirements :
- Bachelor's degree in Computer Science, Engineering, or a related field; advanced degree
- 5 years experience in a data engineering leadership role, managing teams and complex projects.
- Expertise in Databricks (Delta Lake, Apache Spark, Unity Catalog), Snowflake, Azure (Data Factory, Blob), SQL, and Python.
- Expertise in Datorama, Tableau, PowerBI.
- Strong understanding of data modeling, ETL processes, and data warehousing concepts.
- Experience with cloud infrastructure and services, preferably Azure.
- Excellent analytical and problem-solving skills.
- Effective communication and collaboration skills with the ability to work across teams.
29 days ago