Only local San Francisco Bay Area candidates only!
Direct W2 candidates only! No 3rd party agencies!
Hybrid role : 3 days remotely, 2 days onsite
Job Title : Sr Cloud Data Engineer
One of our direct clients is seeking a highly skilled and motivated Cloud Data Engineer to join our growing data team. The ideal candidate will have a strong background in developing data pipelines, implementing data models, and executing best practices for developing data products.
This role requires expertise in AWS, GitLab CI / CD, dbt, Snowflake, SQL, Python, Git, and DevOps, as well as hands-on experience with orchestrators such as Airflow and AutoMateNow, and data governance technologies like Monte Carlo and Collibra.
Key Responsibilities :
Develop Data Pipelines : Design, develop, and maintain robust, scalable, and efficient ETL / ELT pipelines to support diverse data sources and large-scale data processing.
Data Modeling : Create and maintain data models and data architecture to ensure data integrity and optimum performance.
Best Practices Implementation : Apply industry best practices in data warehousing, data governance, and data lifecycle management.
Collaboration : Work closely with data scientists, analysts, and other stakeholders to gather requirements and deliver high-quality data products.
Automation and CI / CD : Implement CI / CD pipelines using GitLab for automated testing, deployment, and integration of data workflows.
Database Management : Manage and optimize Snowflake databases for performance, scalability, and cost-effectiveness.
Coding and Scripting : Write efficient SQL queries and Python scripts to extract, load, and transform data.
Version Control : Utilize Git for version control, ensuring traceable and manageable code changes.
Orchestration : Utilize orchestration tools like Airflow and AutoMateNow to manage and automate data workflows.
Data Governance : Implement and manage data governance and data observability solutions using technologies like Monte Carlo and Collibra.
Monitoring and Optimization : Monitor data pipeline performance and troubleshoot issues to ensure reliability and efficiency.
Documentation : Maintain comprehensive documentation for data pipelines, models, and processes.
Qualifications :
Education : Bachelor s degree in Computer Science, Information Technology, Data Science, or a related field. Experience :
Minimum 5 years of experience in data engineering or a similar role.
Proven experience with AWS cloud services related to data processing, such as S3, Redshift, Lambda, Glue, and Data Pipeline.
Technical Skills :
Proficiency in designing and implementing CI / CD pipelines with GitLab.
Experience with dbt (data build tool) for transforming data within the warehouse.
Strong knowledge of Snowflake with hands-on experience in data warehousing solutions.
Expertise in SQL for data querying, manipulation, and optimization.
Proficiency in Python for scripting and automation tasks.
Familiarity with DevOps practices and tools.
Experience with orchestrators like Airflow and AutoMateNow.
Knowledge of data governance and observability tools such as Monte Carlo and Collibra.
Strong working knowledge of Git for version control. Soft Skills :
Strong problem-solving and analytical skills.
Excellent communication skills, both written and verbal.
Ability to work collaboratively in a fast-paced, dynamic environment.
High attention to detail and commitment to producing high-quality work.
Preferred Qualifications :
AWS Certified Data Analytics Specialty or AWS Certified Solutions Architect.
Experience with real-time data processing and streaming technologies (e.g., Kafka, Kinesis).
Experience in a regulated industry
Experience in a manufacturing environment
Data Engineer
You will be working as a "Data Engineer" with Strong Programming and SQL knowledge. The prospective individual needs to have a good understanding of marketing Domain, Data warehousing , Analytics, Programming and Strong Database Experience. Build and scale data infrastructure that powers batch and r...
Data Engineer
You’ve likely been a data engineer at traditional companies but you’re ready to be the first data hire at a startup. You’re a generalist data and analytics expert who builds data infrastructure at scale. Replicate is a complex business and we need solid data infrastructure to guide it. You’ve set up...
Senior Software Engineer, ML Data Pipeline
Bachelor's degree in Computer Engineering, Computer Science, Electrical Engineering, Robotics or a related field and 4+ years of relevant experience (or Master's degree and 2+ years of relevant experience, or PhD). When you join the Latitude team, you'll work alongside leading experts across machine...
Software engineer- Data Acquisition
We are looking for people with experience gathering data from the web at scale. Experience working with research datasets. ...
Senior Data Engineer
You will work with data scientists, ML pipelines, data processing automation, data processing pipelines, model deployments and experimentation configuration, data quality, data warehousing, data privacy and governance – to name a few. Our data engineers are end-to-end owners who have the opportunity...
Senior Principal Engineer, Data Infra
Data Infrastructure Landscape: In-depth knowledge of modern data infrastructure components, including cloud services, databases (SQL/NoSQL), big data processing frameworks (Spark, Trino or similar), and data management architectures (Hudi, Iceberg or similar). Cardlytics is seeking a Senior Principa...
Data Pipeline Engineer
Koko Home is seeking a Data Pipeline Engineer to develop and maintain data pipelines from our IoT devices all the way to our model building infrastructure. Help filter data and label data by means of heuristics, model-based labeling as well as manual labeling when needed. D in Computer Science, Elec...
Sr. Data Engineer (Platform)
Collaborate with product managers, data scientists, data analysts and engineers to define requirements and data specifications. GoodRx is looking for extremely smart and curious data engineers, who are deft at working with a wide variety of languages, a variety of raw data formats, such as parquet, ...
Data Quality Engineer, Senior
In this role you will be working with Data & Analytics, Data Engineering, Data Analysis, Application, and Business teams. The Data Services Quality Engineering team is part of the BSC Data & Analytics Organization. We provide testing services for Data & Analytics projects that involve data within th...
Software Engineer, Data Platform
You'll work cross-functionally with partners from the Data Science, Data Engineering, AI, Product, Go-to-Market, Legal and Finance organizations to deliver short- and long-term impact. You have worked cross-functionally to establish the right overarching data architecture for a company's needs, to b...