Data Engineer
Newark, NJ
Hybrid
Job Description :
- Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience
- Experience implementing, supporting data lakes, data warehouses and data applications on AWS for large enterprises
- Programming experience with Python, Shell scripting and SQL
- Solid experience of AWS services such as CloudFormation, S3, Athena , Glue, EMR / Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, SM etc.
- Solid experience implementing solutions on AWS based data lakes.
- Should have good experience with AWS Services - API Gateway, Lambda, Step Functions, SQS, DynamoDB, S3, ElasticSearch
- Serverless application development using AWS Lambda
- Experience in AWS data lake / data warehouse / business analytics
- Experience in system analysis, design, development, and implementation of data ingestion pipeline in AWS
- Proven experience in developing, deploying, and optimizing AI / ML models
- Knowledge of ETL / ELT
- End-to-end data solutions (ingest, storage, integration, processing, access) on AWS
- Architect and implement CI / CD strategy for EDP
- Implement high velocity streaming solutions using Amazon Kinesis, SQS, and Kafka (preferred)
- Migrate data from traditional relational database systems, file systems, NAS shares to AWS relational databases such as Amazon RDS, Aurora, and Redshift
- Migrate data from APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift
- Implement POCs on any new technology or tools to be implemented on EDP and onboard for real use-case
- AWS Solutions Architect or AWS Developer Certification preferred
- Good understanding of Lakehouse / data cloud architecture
Responsibilities :
- Designing, building and maintaining efficient, reusable, and reliable architecture and code.
- Build reliable and robust Data ingestion pipelines (within AWS, onprem to AWS ,etc)
- Ensure the best possible performance and quality of high scale data engineering project
- Participate in the architecture and system design discussions
- Independently perform hands on development and unit testing of the applications;
- Collaborate with the development team and build individual components into complex enterprise web systems;
- Work in a team environment with product, production operation, QE / QA and cross functional teams to deliver a project throughout the whole software development cycle;
- Responsible to identify and resolve any performance issues
- Keep up to date with new technology development and implementation
- Participate in code review to make sure standards and best practices are met.
Thanks & Regards
Sai Kiran
Sr Lead Software Engineer - Big Data/ Spark/Hadoop
A motivated individual with a strong background in data platforms, ETL pipeline setup and analysis, you will lead the data catalog initiatives to establish and maintain a cohesive and effective data cataloging system, integral to the broader data governance strategy and objectives. Experience in sof...
Sr. AWS Data Engineer
Senior Data Engineering ETL Developer who will be responsible for designing and implementing robust ETL pipelines, integrating AWS services such as Redshift and S3, and developing telemetry dashboards that provide insightful analytics on developer activities. Document all data engineering procedures...
Field Data Analyst
Serve as primary customer interface to Product Management and Engineering teams for exchange of relevant information. Provide administration support for environmental data acquisition. Knowledge of Windows-based word-processing, spreadsheet, database, and presentation software. SCS Engineers is an E...
Electrical Engineer - Data Center Design
Electrical Engineer - Data Center Design. Company Type: AEC / MEP Engineering, Discipline: Engineering. Our client is an Engineering Design and Commissioning Company that is a leader in the data center space. Create designs and plans, provide and gather specifications, and execute assignments in a m...
Software Engineer, Developer Productivity
We are a team of doers, seasoned engineers, hackers and builders, working on the future of streaming data. Think of it as a streaming data API platform that scales with you from the smallest projects to petabytes of data distributed across the globe. Funded by premier investors including GV and Ligh...
Senior Data Engineer - Enterprise Identity Services
The data operations team will be responsible for data engineer and analysis, quality measures and improvement, data flows and forecasting, and managing our relationship with our customers. This role will join a Data Operations team directly responsible for improving the accuracy and understanding of...
Senior Snowflake Data Engineer
We are looking for Data architect (Modeller) with 10+ Years experience and will be responsible for the overall solution design including high-level data architecture, data model and load patterns. Strong Data Modelling skills and in particular experience with data vault. Experience in architecting S...
Data Engineer
Experience implementing, supporting data lakes, data warehouses and data applications on AWS for large enterprises. Migrate data from traditional relational database systems, file systems, NAS shares to AWS relational databases such as Amazon RDS, Aurora, and Redshift. Migrate data from APIs to AWS ...
Data Engineer
Should have experience using following software/tools:• Big data tools• Relational and NoSQL databases• Data pipeline and workflow management tools• AWS cloud services• Stream processing systems• Ob...
PGIM Private Capital, Data Engineer Manager (Newark, Chicago or Dallas)
Strong proficiency with Azure cloud services including Azure Data Factory, Azure Data Lake, Azure SQL Database, and Azure Databricks preferred. Design and implement highly scalable data ingestion and processing pipelines using Azure Data Factory, Azure Databricks, and other Azure services. Develop a...