Senior Data Engineer

Senior Data Engineer

POSTED : July 5, 2024 LOCATION : Dallas TX POSITION ID : #246038 EMPLOYMENT TYPE : Direct Hire Salary : $92000

• Location : Dallas, TX
• Type : Direct Hire
• Job #246038
• Salary : $92,000

SENIOR DATA ENGINEER

Job Summary :

The Intersect Group is seeking an experienced Data Engineer for our direct banking client, where you will handle the development of all aspects of data processing, including data warehouse architecture / modeling and ETL processing.

You will create and deliver analytical solutions using various tools, collaborate with business stakeholders to collect requirements and translate them into technical specifications and process documentation.

Work on the architecture and development of an event-driven data warehouse, streaming, batch, data modeling, and storage.

Provide technical guidance. Review code and test results. Review pulls requests and scripts for production development. Work on advanced databases, to write and optimize SQL queries, stored procedures, functions, partition data, indexing, and reading execution plans.

Write and debug Python / PySpark scripts to generate extracts, clean, conform, and deliver data for consumption. Implement ETL architecture, data profiling, process flow, metric logic, and error handling.

Support continuous improvement by exploring and presenting alternatives to processes and technologies.

Responsibilities and Duties :

• Enable the design of conceptual architecture and technical solutions.
• Design, develop and maintain scalable and efficient data pipelines and workflows using Azure Synapse to support various business use cases and analytical requirements.
• Collaborate closely with cross-functional teams to understand data requirements and translate them into technical solutions.
• Implement and manage CI / CD pipelines using Azure DevOps to automate the deployment, testing and monitoring of data engineering solutions, ensure continuous integration and delivery.
• Optimize and tune data pipelines and process workflows for performance, scalability, and reliability.
• Leverage best practices and monitoring tools to identify and address bottlenecks and inefficiencies.
• Utilize Azure Data Studio to design and optimize database schemas, stored procedures, and SQL queries.
• Ensure best practices in data integrity, efficiency, and security.
• Develop and maintain ETL (Extract, Transform, Load) processes to ingest, cleanse, transform and load from various sources to target systems.
• Monitor and troubleshoot data pipeline health and performance issues, proactively identifying and resolving issues to minimize downtime and ensure data quality and reliability.
• Create and maintain comprehensive documentation for data pipelines, workflows, and system configurations, ensuring knowledge transfer and compliance with organizational standards and policies.
• Conduct data profiling, cleansing and validation activities to ensure data quality and integrity throughout the data life cycle, adhering to data governance and compliance requirements.
• Participate in code review, testing, and debugging activities to ensure the reliability, stability, and maintainability of data engineering solutions.
• Provide technical guidance, mentorship, and support to team members.
• Troubleshoot and resolve data-related issues reported by end-users or detected through monitoring and alerting systems, ensuring timely resolution and minimal disruption to operations.
• Design and implement a mature data lifecycle, including backups and restore of data.
• Assist ServiceNow admins with creating workflows relevant to the data lifecycle.
• Ensure compliance with Role-Based Access and other regulatory requirements.
• Developing and implementing data governance policies and procedures to ensure data accuracy, completeness, and consistency.
• Designing and implementing data visualization and reporting solutions to enable business users to easily access and analyze data.
• Collaborating with data architects, engineers, and analysts to design, develop, and maintain data pipelines and ETL processes.
• Ensuring data quality and integrity by implementing data validation, cleansing, and testing methods
• Troubleshooting and resolving data issues and providing technical support as needed
• 7 years or more experience required.
• Bachelor's degree required.
5 days ago