Job Description
The Data Engineer will focus on building data pipelines to support our data ingesting, cleansing, enriching, and presentation efforts in support of our flagship SaaS applications.
Responsibilities
- Build data pipelines to support our data ingesting, cleansing, enriching, and presentation efforts in support of our flagship SaaS applications
- Collaborate with our data warehouse, platform, and product teams to build scalable, performant data pipelines.
- Create and extend data models based on changing source data and business requirements.
- Maintain and optimize existing pipelines.
- Automate testing, build, and deployment of data pipelines.
- Participate in code reviews.
- Investigate and diagnose root cause for data warehouse operational issues.
- Function as a technical resource for team members and internal users.
Desired Experience and Skills :
- 2+ years of experience with Python, common packages, and tools.
- Ability to write solid unit tests.
- Familiarity with object-oriented concepts and TDD.
- 3+ years relational database experience and advanced SQL writing and optimization skills.
- Experience with revision control (git), code reviews, iterative / incremental development processes.
- Experience with batch and streaming ETL / ELT.
- Experience with Snowflake (or similar cloud database) and cloud warehouse data modeling.
- Experience / familiarity with AWS services (serverless, S3, SNS, SQS).
- Strong troubleshooting and analytical skills.
- Comfortable working with bash / zsh and shell scripting.
Additional Experience and Skills
- Experience with CI / CD tools (Jenkins, Docker) and build / deploy automation.
- Experience with Airflow pipeline development and AWS MWAA.
- Experience with AWS Analytics services.
- Familiarity with DataDog, Sentry, Prometheus.
Behavioural Competencies :
- Planning and Organization : Able to establish and execute a course of action to accomplish specific short- and long-term goals, to plan appropriate allocation of resources and time, and to approach all responsibilities in an organized manner.
- Driven to Achieve. Able to manage self to drive performance by balancing efforts across the team to exceed targets.
- Initiative : Actively influence events to achieve goals. Acts beyond what is called for; generates and / or recognizes imaginative, creative solutions.
- Adaptability / Flexibility : Able to be effective in many situations and to develop and accept new methods of doing things.
Able to react quickly and to successfully implement changes as needed.
Curiosity : Inquisitive with a thirst for knowledge and desire to learn about the market and all aspects of the business
This position does not provide sponsorship. All applicants should either be US Citizens or Permanent Residents eligible to work in the US without immigration restrictions.
LI-CC1
LI-Remote
Senior Data Engineer - Capital One
Our Authorization Data Pipeline Platform is an AWS Glue ( Java flavor ) based batch processing framework designed to efficiently process data from our enterprise data lake, perform necessary transformations, ensure data quality, and load the results into Authorization Database tables in order to ena...
Senior Data Engineer
Hands on experience on data profiling, data modeling and data engineering using relational databases like Snowflake, Oracle, SQL Server; ETL tools like Informatica IICS; scripting using Python, R, or Scala; workflow management tools like Autosys. Demonstrate high proficiency across a wide range of t...
Lead Data Engineer
New York City (Hybrid On-Site): $201,400 - $229,900 for Lead Data EngineerIllinois (Hybrid On-Site): $187,700 - $214,200 for Lead Data EngineerCandidates hired to work in other locations will be subject to the pay range associated with that location, and the actual annualized salary amount offered t...
Senior Data Engineer
Leverage the Databricks Data Intelligence Platform to design and implement innovative data solutions, including lakehouse architectures built that enable seamless data access and advanced analytics. Utilize SQL, Python, and PySpark to extract, transform, and integrate diverse customer & vendor data,...
Lead Data Engineer (Spark, Scala, SQL) - Capital One
This is a team of skilled data engineers that build robust data pipelines, ensuring data integrity accuracy and accessibility for all stakeholders. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. West Wacker Dr (35012), United States of America, Chica...
Lead Data Engineer (python, Spark, Big Data, AWS)
West Wacker Dr (35012), United States of America, Chicago, IllinoisLead Data Engineer (python, Spark, Big Data, AWS)Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment...
Big Data Engineer
Big Data Engineer Job Description Reporting to the Lead Technical Architect for our Commercial Data Lake for Finance (CDL-F), the Big Data Engineer will be responsible for the technical development of data, analytics, and GenAI solutions in support of the US Therapeutics Finance business unit. Big D...
Lead Data Engineer (Python, Spark, AWS)
We are seeking Data Engineers who are passionate about marrying data with emerging technologies. West Wacker Dr (35012), United States of America, Chicago, IllinoisLead Data Engineer (Python, Spark, AWS)Do you love building and pioneering in the technology space? Do you enjoy solving complex busines...
Data Engineer - (Contract-To-Hire) REMOTE
Salesforce Data Engineer will partner with other team members on the creation of best-practices approach to capture data from disparate applications into an Azure Data Lake for cleaning, curation and mapping into various application destinations meeting Gallagher approved standards. Salesforce Data ...
Distinguished Engineer, Director - Card Core Data Architecture
About the team:Card Core Data Architecture team is modernizing the entire data ecosystem of Capital One’s credit card core through data standardization and build out of Data Products across Streaming, API, data Lake and data warehouse platforms. As a member of the data architecture team, you will ha...