Sr Data Engineer

The Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.

Responsibilities

• Data Solutions : Design and implementation of data pipelines using GCP services.
• Manage Deployments with Helm and Kubernetes : Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management, ensuring efficient orchestration of services.
• Automation with Jenkins : Implement CI / CD pipelines using Jenkins to automate data pipeline deployment, testing, and integration with other services, ensuring quick iterations and deployments.
• Develop and Optimize Pipelines : Write efficient Python and SQL scripts to build data pipelines and ETL / ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.
• Data Integration and Orchestration : Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow).
• Security and Compliance : Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in GCP, adhering to best practices for handling sensitive data.
• Collaborate Across Teams : Work closely with application developers, data architects, and business stakeholders to define and deliver robust data-driven solutions. Provide technical leadership and ensure alignment between business and technical objectives.
• Monitoring and Logging : Set up monitoring, logging, and alerting using Cloud Monitoring (formerly Stackdriver), Datadog, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.

Skills

J-18808-Ljbffr