Description : Summary
Summary
The Data Engineer will be responsible for building, maintaining data pipelines and data products to ingest, process large volume of structured / unstructured data from various sources.
The Data engineer will work on analyzing the data needs, migrating the data into an Enterprise data lake, build data products and reports.
The role requires experience with building real time and batch based ETL pipelines with strong understanding of big data technologies and distributed processing frameworks with.
Skill Needs
oExpertise working with large scale distributed systems (Hadoop, Spark).
oStrong understanding of the big data cluster, and its architecture
oExperience building and optimizing big data ETL pipelines.
oAdvanced programming skills with Python, Java, Scala
oGood knowledge of spark internals and performance tuning of spark jobs.
oStrong SQL skills and is comfortable operating with relational data models and structure.
oCapable of accessing data via a variety of API / RESTful services.
oExperience with messaging systems like Kafka.
oExperience with No SQL databases. Neo4j, mongo, etc.
oExpertise with Continuous Integration / Continuous Delivery workflows and supporting applications.
oExposure to cloud environments and architectures. (preferably Azure)
oAbility to work collaboratively with other teams.
oExperience with containerization using tools such as Docker.
oStrong knowledge of Linux and Bash. Can interact with the OS at the command line and create shell scripts to automate workflows.
oAdvanced understanding of software development and collaboration, including experience with tools such as Git.
oExcellent written and verbal communication skills, comfortable presenting in front of non-technical audiences.
Essential Responsibilities include but not limited to :
oDesign and develop ETL workflows to migrate data from varied data sources including SQL Server, Netezza, Kafka etc. in batch and real-time.
oDevelop checks and balances to ensure integrity of the ingested data.
oDesign and Develop Spark jobs as per requirements for data processing needs.
oWork with Analysts and Data Scientists to assist them in building scalable data products.
oDesigns systems, alerts, and dashboards to monitor data products in production
Data Engineer
Have experience with Data Modelling, Data integration, Data Quality Management. Elsdon Technology are proud to be partnered with a company who are looking to make 2 hires over the coming weeks for a mid and lead engineer. ...
Data Engineer/ Business Intelligence (2551403)
As a Data/BI Engineer, you will play a key role in developing and maintaining our data infrastructure, expanding Data Lake and Data Warehouse instances, and building robust ELT pipelines using MS Fabric and SQL Server. Work closely with stakeholders, data analysts, and third-party data engineers to ...
Data Engineer
The Data engineer will work on analyzing the data needs, migrating the data into an Enterprise data lake, build data products and reports. The Data Engineer will be responsible for building, maintaining data pipelines and data products to ingest, process large volume of structured / unstructured dat...
LLM Data Engineer | United States | Fully Remote
We are seeking an experienced AI/LLM Data Engineer to build and maintain the data pipeline for our Generative AI platform. The ideal candidate will be well-versed in the latest Large Language Model (LLM) technologies and have a strong background in data engineering, with a focus on Retrieval-Augment...
Senior Data Engineer
Advanced knowledge of database and data warehousing concepts, including data lakes, relational and dimensional database design concepts, and data modeling practices --- knows how to use inmon or Kimball on building data warehousing dimensional modeling (, building a data mart early on in their caree...
Data Engineer (SSIS Experience) - US Based
Role: Data Engineer (Mid-Level) Location: Remote (US-based candidates only) Position Type: Contract - 4 weeks (Short-term engagement) Start Date: October 21st Position Summary Data Meaning is a Business Intelligence consulting company. Solid understanding of data modeling, data warehousing, a...
Data Engineer
Investment Data Platform within FIS Capital Markets is aimed at providing a simplified data experience for our clients by providing a platform to aggregate investment data (data warehouse), visualize reports and analytics, participate in data marketplace (data marts), all while governing and orchest...
Staff Software Engineer, Data - Level 4
Develop data quality measurement and monitoring systems, metadata management, data catalogs, and Master Data Management (MDM) solutions. The Data team's mission is to build big data technologies, platforms, systems, and tools that clean, process, enrich, and optimize core company data, making it eas...
Lead, Software Engineer Data Management
As a Lead Software Engineer on/in Data Management & Governance you will partner with product owners, tech leads, designers, engineers and delivery professionals to improve Data Management and Governance services. Experience in building scalable and stronger data pipelines to support data integra...
Analytic Data Engineer – Databricks
This role within the 'Data Engineering & Self-Service Products' team involves architecting and engineering analytic data solutions, including designing and developing data marts in Databricks using PySpark or Spark SQL, building interactive Power BI dashboards to visualize KPIs and trends, and creat...