Job Description
Job Description
We are looking for a passionate certified Data Engineer. The successful candidate will turn data into information, information into insight and insight into business decisions.
Data analyst responsibilities include conducting full lifecycle analysis to include requirements, activities and design. Data analysts will develop analysis and reporting capabilities.
They will also monitor performance and quality control plans to identify improvements.
Primary skillset : Python / Pyspark & Azure, ADF, SQL, SQL Server, Data Warehousing, ETL
Secondary : Databricks
Nice to have : Informatica / ETL.
Responsibilities
- Design and develop ETL processes based on functional and non-functional requirements in python / pyspark within Azure platform
- Understand the full end to end development activities from design to go live for ETL development and Azure platform
Recommend and execute improvements
- Document component design for developers and for broader communication.
- Understand and adopt an Agile (SCRUM like) software development mindset
- Follow established processes / standards, business technology architecture for development, release management and deployment process
- Execute and provide support during testing cycles and post-production deployment, engage in peer code reviews.
- Elicit, analyze, interpret business and data requirements to develop complete business solutions, includes data models (entity relationship diagrams, dimensional data models), ETL and business rules, data life cycle management, governance, lineage, metadata and reporting elements.
- Apply automation and innovation on new and on-going data platforms for those development projects aligned to business or organizational strategies.
- Design, develop and implement reporting platforms (e.g. modeling, ETL, BI framework) and complex ETL frameworks that meet business requirements.
- Deliver business or enterprise data deliverables (that adhere to enterprise frameworks) for various platforms / servers / applications / systems.
Requirements
- Proven working experience as a data engineer
- Bachelor degree or equivalent in Computer Science
- Skilled in Python object-oriented programming
- Skilled in AWS Compute such as EC2, Lambda, Beanstalk, or ECS
- Skilled in AWS Database products such as Neptune, RDS, Redshift, or Aurora
- Skilled in AWS Management and Governance suite of products such as CloudTrail, CloudWatch, or Systems Manager
- Skilled in Amazon Web Services (AWS) offerings, development, and networking platforms
- Skilled in SQL
- Skilled in Jenkins
- Skilled in JSON
- Skilled in discovering patterns in large data sets with the use of relevant software such as Oracle Data Mining or Informatica
- Skilled in cloud technologies and cloud computing
- Experience using software and computer systems' architectural principles to integrate enterprise computer applications such as xMatters, AWS Application Integration, or WebSphere
- Determining causes of operating errors and taking corrective action
- Experience in the process of analyzing data to identify trends or relationships to inform conclusions about the data
- Skilled in creating and managing databases with the use of relevant software such as MySQL, Hadoop, or MongoDB
- Programming including coding, debugging, and using relevant programming languages
- Communication including communicating in writing or verbally, copywriting, planning and distributing communication, etc.
- Ability to frame ideas as systems and analyzing the inputs, outputs, and process
- Experience helping an organization to plan and manage change in effort to meet strategic objectives
- Adept at managing project plans, resources, and people to ensure successful project completion
- Working with people with different functional expertise respectfully and cooperatively to work toward a common goal
Junior Data Engineer
Key Responsibilities:Build and support data migration staging environments and warehousesProfile downstream data to identify anomalies and potential issuesCreate scripts for reporting and collaborate with QA to develop test casesRequired Qualifications and Education:Master's in computer science or B...
Lead Data Engineer
We are seeking Data Engineers who are passionate about marrying data with emerging technologies. Locations: VA - McLean, United States of America, McLean, VirginiaLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-p...
Senior Data Visualization Engineer
A company is looking for a Senior Manager, Data Visualization Engineer to lead a team responsible for building and maintaining data visualization solutions. ...
Senior Data Engineer (Spark, Kafka,Flink)
Team Info:We are building a platform to enable engineers and data analysts to quickly deploy real time streaming data transformation jobs within the Capital One data ecosystem. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. This will remove the need ...
Data Warehouse Engineer
A company is looking for a Data Warehouse Engineer to work 100% remote with an active secret clearance. ...
Distinguished Engineer, Director - Card Core Data Architecture
About the team:Card Core Data Architecture team is modernizing the entire data ecosystem of Capital One’s credit card core through data standardization and build out of Data Products across Streaming, API, data Lake and data warehouse platforms. As a member of the data architecture team, you will ha...
Senior Data Engineer
Salary budget: $125,000 gross per year + benefits....
Staff Software Engineer - Data Infrastructure/Data Products
The Data Infrastructure team is looking to hire a Staff software Engineer who is excited to solve large scale batch and streaming data challenges. Our community of users generates over B analytics events per day, each of which is ingested by the Data Infrastructure team into a data warehouse that se...
Senior Data Engineer
We are seeking Data Analysts who are not only technically adept, but also understand the importance of harnessing the power of data to streamline financial operations, enhance decision-making and improve business outcomes. Maintain seamless data links between internal systems and ApptioOne, ensuring...
Senior Big Data Engineer
In partnership with other business, platform, technology, and analytic teams across the enterprise, design, build and maintain well-engineered data solutions in a variety of environments, including traditional data warehouses, Big Data solutions, and cloud-oriented platforms. Align with security, da...