Data Engineering Manager

We are hiring an data engineering manager on the team that is building a technology platform that spans our client's businesses, enabling new capabilities.

The technology team maintains the firm’s cloud infrastructure, multiple application platforms, development tooling, and data architecture.

The manager will contribute to the design and implementation of these systems and serve as a leader on this team. We are looking for someone who values teamwork and has the desire to mentor other data engineers.

We need an engineer to shape the vision for the data platform as well as build the software, processes, and tools needed.

In addition, this team is responsible for building and maintaining common libraries and data frameworks. These libraries provide business applications with core functionalities needed to deliver data for the front, middle, and back offices.

This role will also oversee data engineering including designing, implementing, and maintaining scalable data pipelines.

The existing pipeline repository, data storage architecture, and data access APIs need to be enhanced and standardized. There is extensive work related to the automation and support for the Extract, Transform, and Load (ETL) processes from external data sources.

In addition, we want to provide cloud-based runtimes with the necessary tooling to support internal applications. This role will work with several teams to ensure standardized technologies are adopted across the firm.

The position provides the opportunity to work on establishing the core functionality of our enterprise development platform, which provides the foundational components utilized by applications throughout, including web deployment and workflow orchestration.

This is a hands-on role, and candidates should have had experience deploying and supporting production code in cloud environments as well as automating the build and management of a range of cloud environments.

It is critical that candidates have familiarity with testing and have experience implementing CI / CD tools with infrastructure as code using terraform.

Deep knowledge of containerization, networking, security, cross system communication, configuration management, logging, metrics, monitoring, and alerting is important.

Another key responsibility for this role will be to streamline and oversee the SDLC process used by developers to bring projects from inception through to production and establish a strong devops mindset.

The successful candidate will support the evolution of the firm’s technological capabilities by improving all aspects of the development lifecycle and maintaining high standards for code quality and organization.

Responsibilities :

• Deliver the next iteration version of our internal technology platform
• Architect, develop and maintain platform components, frameworks, and offerings
• Build infrastructure to make data sets accessible and useful
• Develop tools and processes for acquisition, validation, organization, and storage of large volumes of structured and unstructured data from many sources
• Develop systems to improve and test data quality
• Support and educate team members on best practices in software engineering and participate in code reviews
• Establish a strong software engineering culture and high code quality standards.
• Produce documentation for technical and non-technical audiences
• Provide leadership for the team by providing feedback on code, behavioral challenges, and other work items
• Manage systems efficiently at scale through standardization, automation, testing, and monitoring
• Enforce standards for all phases of software development cycle including source control and testing
• Work with business owners to regularly re-prioritize projects while delivering both tactical and long-term objectives
• Develop well defined processes for integrating new technologies to meet both business and technical needs
• Drive the elimination of operational tasks through automation and self-service
• Drive efficient utilization of resources through metrics and reporting
• Manage multiple environments and promote code between them
• Implement standardized methodologies and CI / CD tooling that automate testing, builds, and deployment
• Architect, develop and maintain solutions for business teams and help them do the same

Required Qualifications :

• Strong background in software and data engineering (10+ years)
• Strong Python knowledge including OOP
• Broad knowledge and experience with relational databases (stored procedures, indexing, partitioning, table design, query optimization, and design) with proficiency in SQL
• Exposure to data warehouses (snowflake, redshift)
• Strong knowledge of cloud environments (preferably AWS, ideally exposure to both windows and Linux), including security, networking, and infrastructure as code (Terraform)
• Experience with automation / orchestration platforms (Airflow)
• Extensive experience with version / source control systems (Git)
• Exposure to commodities and weather markets, including electricity, congestion markets, natural gas, oil, and emissions and related data sets (ISO APIs, fundamental data vendors, government APIs).
• Strong financial and fundamental data knowledge and experience working with large datasets
• Ability to execute independently, on a deadline, and under pressure
• Experience with deploying monitoring and alerting solutions
• Experience with designing and implementing data pipelines for quality, acquisition, validation, and related infrastructure
• Communication skills and project management experience
• Excellent troubleshooting and analytical skills
• Experience in data visualization and web user interfaces is a plus
• Exceptional attention to detail
• Ability to understand unfamiliar code bases (multi and mono repo structure) and debug applications
• Experience managing a team of developers
• Experience designing and architecting libraries
• Knowledge of derivative pricing, products, and markets
• Consistent track record managing and balancing priorities across a diverse set of stakeholders and guiding them toward strategic solution
29 days ago