Data Engineer

APN Consulting has an immediate need for a direct client requirement :

Direct Client Requirement

Fulltime / Contract

Multiple Location - Remote Option available

Job Title : Data Engineer

Job Summary :

We are seeking a skilled Data Engineer to join our team. The ideal candidate will have experience in building and optimizing data pipelines, architectures, and data sets.

You will support our software developers, database architects, data analysts, and data scientists on data initiatives and ensure optimal data delivery architecture is consistent throughout ongoing projects.

You must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products.

Key Responsibilities :

Design, build, and maintain data pipelines : Develop and optimize ETL (Extract, Transform, Load) processes to extract data from various sources, transform it into a usable format, and load it into databases or data warehouses.

Data architecture and management : Develop and maintain scalable data architectures, including databases, data warehouses, and data lakes.

Implement strategies for data quality, integrity, security, and retention.

Collaborate with teams : Work closely with data scientists, analysts, and other stakeholders to understand data needs and deliver solutions that meet those needs.

Optimize performance : Monitor and troubleshoot performance issues in data pipelines and databases, optimizing them for efficiency and scalability.

Implement data governance : Ensure that data governance and best practices are followed, including data security, privacy, and compliance standards.

Support data initiatives : Provide support for data analytics and business intelligence initiatives, including data extraction, preparation, and analysis.

Automation and tools : Develop automation tools to improve the efficiency of data processing and reduce manual work.

Documentation : Create and maintain documentation for data architecture, pipelines, and processes.

Required Qualifications :

Education : Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.

Experience :

Proven experience as a Data Engineer or in a similar role.

Experience with big data tools : Hadoop, Spark, Kafka, etc.

Experience with relational SQL and NoSQL databases.

Experience with data pipeline and workflow management tools : Azkaban, Luigi, Airflow, etc.

Technical Skills :

Proficient in programming languages such as Python, Java, or Scala.

Expertise in SQL and database design.

Experience with cloud services such as AWS, Google Cloud, or Azure.

Familiarity with data modeling, ETL / ELT, and data warehousing solutions.

Soft Skills :

Strong problem-solving skills and attention to detail.

Ability to work independently and as part of a team.

Excellent communication and collaboration skills.

Preferred Qualifications :

Experience with machine learning frameworks and tools.

Familiarity with real-time data processing.

Knowledge of containerization and orchestration tools (Docker, Kubernetes).