job summary : undefined
undefined
responsibilities :
The person on this role will play a crucial role in building scalable and cost-effective data pipelines, data lakes, and analytics systems.
- Data Ingestion : Implement data ingestion processes to collect data from various sources, including databases, streaming data, and external APIs.
- Data Transformation : Develop ETL (Extract, Transform, Load) processes to transform and cleanse raw data into a structured and usable format for analysis.
- Data Storage : Manage and optimize data storage solutions, including Amazon S3, Redshift, and other AWS storage services.
- Data Processing : Utilize AWS services like AWS Glue, Amazon EMR, and AWS Lambda to process and analyze large datasets.
- Data Monitoring and Optimization : Continuously monitor and optimize data pipelines and infrastructure for performance, cost-efficiency, and scalability.
- Integration : Collaborate with data scientists, analysts, and other stakeholders to integrate AWS-based solutions into data analytics and reporting platforms.
- Documentation : Maintain thorough documentation of data engineering processes, data flows, and system configurations.
- Scalability : Design AWS-based solutions that can scale to accommodate growing data volumes and changing business requirements.
- Cost Management : Implement cost-effective solutions by optimizing resource usage and recommending cost-saving measures.
- Troubleshooting : Diagnose and resolve AWS-related issues to minimize downtime and disruptions.
- Qualifications
qualifications :
- Experience level : Experienced
- Minimum 9 years of experience
- Education : Bachelors
skills :
- AWS Cloud Formation
- Python
30+ days ago