Sr. Data Engineer

Haddon Heights, New Jersey, United States

Job Overview :

Were seeking an experienced Senior Data Engineer specializing in Azure and Databricks to drive data engineering initiatives, create scalable data pipelines, and support our Enterprise Data Lake and Data Warehouse (Synapse). The ideal candidate will be well-versed in Azure Data Factory, Databricks (DLT), and have hands-on experience with PySpark, SQL, and Azure DevOps. This role requires deep technical knowledge in data pipeline orchestration, data transformation, and optimizing performance within Azure cloud services.

You must reside in the US; No 3rd parties.

Responsibilities :

• Data Pipeline Development : Build new data pipelines and enhance existing ones using Databricks (Data Engineering, Delta Lake), Azure Data Factory, and Synapse to load data into the Enterprise Data Lake, Delta Lake, and Enterprise Data Warehouse.
• Data Transformation and Loading (ETL / ELT) : Implement complex data transformations to parse, cleanse, and load data according to business requirements and best practices.
• Orchestration and Automation : Utilize Azure Data Factory to orchestrate data flows and automate data ingestion processes.
• Testing and Quality Assurance : Perform unit testing, coordinate integration testing, and manage User Acceptance Testing (UAT) to ensure high-quality data.
• Documentation and Runbooks : Develop and maintain high-level design (HLD), detailed design (DD), and runbooks to support production operations and improve data pipeline transparency.
• Compute Configuration and Maintenance : Configure and optimize compute resources, apply Data Quality (DQ) rules, and manage regular maintenance schedules.
• Performance Tuning : Identify and implement performance improvements for data pipelines, ensuring efficiency and scalability.
• Production Support : Provide ongoing support for production environments, troubleshoot issues, and ensure optimal system performance.

Technical Skills :

Qualifications :

J-18808-Ljbffr