Data Engineer ID43228

Job Description

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI / ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US

If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

As a Middle Data Engineer , you’ll build and optimize ETL pipelines and cloud data solutions that provide reliable insights for data scientists and analysts. You’ll tackle complex data challenges, collaborate with cross-functional teams, and grow your expertise in Python, Airflow, Spark, and AWS in a dynamic, innovative environment.

WHAT YOU WILL DO

• Build and support ETL pipelines;
• Monitor data pipelines, identify bottlenecks, optimize data processing and storage for performance and cost-effectiveness;
• Collaborate effectively with cross-functional teams including data scientists, analysts, software engineers, and business stakeholders;
• Working with Terraform to build AWS infrastructure;
• Analyze sources and build Cloud Data Warehouse and Data Lake solution.

MUST HAVES

NICE TO HAVES

PERKS AND BENEFITS

Requirements

3+ years of professional experience with Python; 3+ years of professional experience in a Data Engineering role; Proficiency in programming languages commonly used in data engineering such as Python, SQL, and optionally Scala for working with data processing frameworks like Spark and libs like Pandas; Proficiency in designing, deploying, and managing data pipelines using Apache Airflow for workflow orchestration and scheduling; Ability to design, develop, and optimize ETL processes to move and transform data from various sources into the data warehouse, ensuring data quality, reliability, and efficiency; Knowledge of big data technologies and frameworks such as Apache Spark for processing large volumes of data efficiently; Extensive hands-on experience with various AWS services relevant to data engineering, including but not limited to Amazon MWAA, Amazon S3, Amazon RDS, Amazon EMR, AWS Lambda, AWS Glue, Amazon Redshift, AWS Data Pipeline, Amazon DynamoDB; Deep understanding and practical experience in building and optimizing cloud data warehousing solutions; Ability to monitor data pipelines, identify bottlenecks, and optimize data processing and storage for performance and cost-effectiveness; Excellent communication skills to collaborate effectively with cross-functional teams including data scientists, analysts, software engineers, and business stakeholders; Bachelor’s degree in computer science / engineering or other technical field, or equivalent experience; Upper-intermediate English level.