Context
SII Group USA is recruiting a Senior Data Engineer to start a new project.
The Data Platform is implemented under Google Cloud Platform, with Databricks is used as the Data processing hub.
Improvements targeted :
- Better data administration,
- Get data lineage and availability of additional Databricks features
- Implementation of Databricks Unity Catalog feature.
- migration of data from a single default cloud storage bucket to data-source specific buckets in Google Cloud storage.
Global organization with product engineering teams split between USA and EMEA.
Main Tasks to achieve :
In full autonomy, and as an Data Engineering Expert, you will have to :
- Review current design & propose design updates if required
- Create buckets for each component and environment in Terraform
- Create Meta store and Catalogues in Terraform and assign buckets
- Create users / groups / service principals in Terraform and assign access rights
- Create new ddl and table library for unity catalogue structure
- Migrate to shared clusters and refactor get notebook context info
- Update all notebooks to use new ddl and table library
- Complete validation of unity catalogue on Dev - updated existing unitests - new unit tests added for new notebooks - integration tests
- Data migration workflow with result report will migrate existing data from existing Delta Tables (saved in Databricks system buckets) to Delta Tables saved in external locations (specific buckets)
Senior Software Engineer Big Data Tooling & API Development
Wells Fargo is seeking a Senior Big Data Engineer in the role of a Big Data Tooling & API Development Software Engineer within its Risk Development organization. The Risk Data management Services group is seeking a Senior Big Data Engineer (Sr. Specialty Software Engineer) to work on building and su...
Data Engineer Java Team Lead Senior Vice President
The Data Engineer Team Lead Java Senior Vice President is a senior management level position responsible for accomplishing results through the management of a team or department to establish and implement new or revised application systems and programs in coordination with the Technology Team. In-de...
Senior Performance Data Engineer
Performance & Test Data Engineer (Remote). ...
Senior Engineer, Master Data Management
We are looking for a highly skilled Senior Developer with deep expertise in Master Data Management (MDM) platforms. Implement methodologies for ensuring data accuracy and quality, along with reporting metrics to enhance data integrity. Utilize data integration tools and pipelines to manage the flow ...
Senior Data Visualization Engineer
Software Engineer (Qlik) to design, develop, and maintain key reporting tools. ...
Senior Civil Engineer - Data Center
As a Senior Civil Engineer on our Data Center Civil Team, you will be a part of the firm’s largest and most complex projects. Olsson currently has several opportunities for a Senior Civil Engineer to support our Data Center Site Design group. You will serve as a project manager on some projects and ...
Senior Software Engineer Big Data Tooling & API Development
Wells Fargo is seeking a Senior Big Data Engineer in the role of a Big Data Tooling & API Development Software Engineer within its Risk Development organization. The Risk Data management Services group is seeking a Senior Big Data Engineer (Sr. Specialty Software Engineer) to work on building and su...
Senior Cloud Data Engineer - Remote / Telecommute
Proficiency in tools and technologies: BigQuery, Cloud Functions, Cloud Run, Dataform, Dataflow, Dataproc, SQL, Python, Airflow, PubSub. Proficiency in building end to end data platforms and data services in GCP is a must. Data Fabric - Big Data Processing - Apache Spark. Prior experience architecti...
Senior Data Engineer
Data Integration and ETL: Develop and maintain advanced Data Engineering processes and patterns using Spark, Python, Scala, Databricks, Azure Data Factory and other relevant tools to integrate data from various sources. Reporting to the Data Engineering Manager, the Sr Data Engineer is a software en...
Senior Data Engineer
Design, build and launch efficient and reliable data pipelines to move data from source platforms, including front-end applications, back-end systems, and third-party analytics and data services, to our enterprise data hub. Senior Data Engineer - Dallas, TX. Senior Data Engineer (Sourcing & Pipeline...