Senior Data Engineer

Qualifications

10+ years of experience in data engineering with an emphasis on data analytics and reporting.

6+ years of experience with the cloud platforms Amazon Web Services (AWS), Azure.

2+ years of experience in building data pipelines to support AI and ML models, strong understanding of LLM's, prompt engineering, vibe coding using the GitHub copilot or similar AI tools, helping team to build faster using the enterprise approved AI tools.

10+ years of experience in SQL, data transformations, ETL / ELT and experience on Database Platform Snowflake ,Fabric and S3.

10+ years of experience in the design and build of data extraction, transformation, and loading processes by writing custom data pipelines.

6+ years of experience with one or more of the following scripting languages : Python, SQL, Shell Scripting.

6+ years of experience designing and building real-time data pipelines utilizing various Cloud services such as S3, Kinesis, RDS, Snowflake, Lambda, Glue, API Gateway, SQS, SNS, CloudWatch, cloud formation, DBT, etc.

5+ years of experience in REST API data integrations in pulling into Snowflake, posting from Snowflake, understanding of webhooks, and implementing using AWS gateway.

Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline, or equivalent experience.

4+ years of experience in data modeling to support descriptive analytics in PBI.

4+ years of experience in data cataloging, empowering business, and digital partners to utilize data cataloging and metadata for self-service.

Strong understanding of data management principles and best practices.

understanding of data modeling, including conceptual, logical, and physical data models.

Responsibilities

Translate business requirements into technical specifications; establish and define details, definitions, and requirements of applications, components, and enhancements.

Participate in project planning; identifying milestones, deliverables, and resource requirements; track activities and task execution.

Generate design, development, test plans, detailed functional specifications documents, user interface design, and process flow charts for execution of programming.

Develop data pipelines / APIs using Python, SQL, potentially Spark, and AWS, Azure, or GCP Methods.

Use an analytical, data-driven approach to drive a deep understanding of fast-changing business.

Build large-scale batch and real-time data pipelines with data processing frameworks in AWS, Snowflake, and DBT.

Move data from on-prem to cloud and cloud data conversions.

Implement large-scale data ecosystems including data management, governance, and the integration of structured and unstructured data to generate insights leveraging cloud-based platforms.

Leverage automation, cognitive, and science-based techniques to manage data, predict scenarios, and prescribe actions.

Drive operational efficiency by maintaining their data ecosystems, sourcing analytics expertise, and providing As-a-Service offerings for continuous insights and improvements.

Job Description

Translate business requirements into technical specifications; establish and define details, definitions, and requirements of applications, components, and enhancements.

Participate in project planning; identifying milestones, deliverables, and resource requirements; track activities and task execution.

Generate design, development, test plans, detailed functional specifications documents, user interface design, and process flow charts for execution of programming.

Develop data pipelines / APIs using Python, SQL, AWS, Azure Fabric.

Use an analytical, data-driven approach to drive a deep understanding of fast-changing business.