AI Big Data Engineer

Summary :

As an AI Big Data Engineer, you will focus on creating a Unified Data Platform. You will design, develop, and maintain data pipelines, data lakes, and data platforms to support analytics and business intelligence needs.

This role involves working with cutting-edge technologies and tools like Spark, Kafka, AWS, Azure, and Kubernetes to manage large-scale and complex data challenges.

Collaboration with full stack developers, data scientists, analysts, and stakeholders are essential to ensure data quality, reliability, and usability.

Comfort with handling huge datasets is a must.

Main Responsibilities :

• Build automated pipelines to extract and process data from legacy platforms (predominantly SQL Server), using tools like stored procedures, Glue processing, etc.
• Implement data-related business logic on modern data platforms, such as AWS Glue, Databricks, and Azure, using best practices and industry standards.
• Create vector databases, data marts, and data models to support them.
• Optimize and monitor performance, reliability, and security of data systems and processes.
• Integrate and transform data from various sources and formats, such as structured, unstructured, streaming, and batch.
• Develop and maintain data quality checks, tests, and documentation.
• Support data analysis, reporting, and visualization using tools like SQL, Python, Tableau, and Quicksight.
• Research and evaluate new data technologies and trends to enhance data solutions and capabilities.

Qualifications and Skills :

• Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related field.
• At least 5 years of experience in data engineering or a similar role (DBA experience is a plus).
• Experience with big data frameworks and tools such as Spark, Hadoop, Kafka, and Hive.
• Expert in SQL, with knowledge of efficient query and schema design, DDL, data modeling, and stored procedures.
• Proficient in at least one programming language such as Python, Go, or Java.
• Experience with CI / CD, containerization (e.g., Docker, Kubernetes), and orchestration (e.g., Airflow).
• Experience building production systems with modern ETL / ELT and data platforms such as AWS Glue, Databricks, Snowflake, and Azure Cognitive Search.
• Experience deploying data infrastructure on cloud platforms (AWS, Azure, or GCP).
• Strong knowledge of data quality, governance, and security principles and practices.
• Excellent communication, collaboration, and problem-solving skills.

Desired Skills and Experience

Qualifications and Skills :

Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related

field.

At least 5 years of experience in data engineering or a similar role (DBA experience is

a plus).

• Experience with big data frameworks and tools such as Spark, Hadoop, Kafka, and Hive.
• Expert in SQL, with knowledge of efficient query and schema design, DDL, data modeling,

and stored procedures.

• Proficient in at least one programming language such as Python, Go, or Java.
• Experience with CI / CD, containerization (e.g., Docker, Kubernetes), and orchestration

e.g., Airflow).

Experience building production systems with modern ETL / ELT and data platforms such as

AWS Glue, Databricks, Snowflake, and Azure Cognitive Search.

• Experience deploying data infrastructure on cloud platforms (AWS, Azure, or GCP).
• Strong knowledge of data quality, governance, and security principles and practices.
• Excellent communication, collaboration, and problem-solving skills.

Beacon Hill is an Equal Opportunity Employer that values the strength diversity brings to the workplace. Individuals with Disabilities and Protected Veterans are encouraged to apply.

If you would like to complete our voluntary self-identification form, please click here or copy and paste the following link into an open window in your browser :

Completion of this form is voluntary and will not affect your opportunity for employment, or the terms or conditions of your employment.

This form will be used for reporting purposes only and will be kept separate from all other records.

Company Profile :

Beacon Hill Technologies, a premier National Information Technology Staffing Group, provides world class technology talent across all industries utilizing a complete suite of staffing services.

Beacon Hill Technologies' dedicated team of recruiting and staffing experts consistently delivers quality IT professionals to solve our customers' technical and business needs.

Beacon Hill Technologies covers a broad spectrum of IT positions, including Project Management and Business Analysis, Programming / Development, Database, Infrastructure, Quality Assurance, Production / Support and ERP roles.

Learn more about Beacon Hill and our specialty divisions, Beacon Hill Associates, Beacon Hill Financial, Beacon Hill HR, Beacon Hill Legal, Beacon Hill Life Sciences and Beacon Hill Technologies by visiting .

We look forward to working with you.

Beacon Hill. Employing the Future™