REQT : Direct Client : Big Data Engineer @ Austin, TX – REMOTE - Locals Only

Big Data Engineer

Austin, TX REMOTE

8 months Contract

We are looking for candidates who can work on our W2.

Candidates with prior experience working for State Government clients are preferred.

The Worker is responsible for developing, maintaining, and optimizing big data solutions using the Databricks Unified Analytics Platform.

This role supports data engineering, machine learning, and analytics initiatives within this organization that relies on large-scale data processing.

Duties include :

• Designing and developing scalable data pipelines
• Implementing ETL / ELT workflows
• Optimizing Spark jobs
• Integrating with Azure Data Factory
• Automating deployments
• Collaborating with cross-functional teams
• Ensuring data quality, governance, and security.

II. CANDIDATE SKILLS AND QUALIFICATIONS

Minimum Requirements :

Candidates that do not meet or exceed the minimum stated requirements (skills / experience) will be displayed to customers but may not be chosen for this opportunity.

Years

Required / Preferred

Experience

Required

Implement ETL / ELT workflows for both structured and unstructured data

Required

Automate deployments using CI / CD tools

Required

Collaborate with cross-functional teams including data scientists, analysts, and stakeholders

Required

Design and maintain data models, schemas, and database structures to support analytical and operational use cases

Required

Evaluate and implement appropriate data storage solutions, including data lakes (Azure Data Lake Storage) and data warehouses

Required

Implement data validation and quality checks to ensure accuracy and consistency

Required

Contribute to data governance initiatives, including metadata management, data lineage, and data cataloging

Required

Implement data security measures, including encryption, access controls, and auditing; ensure compliance with regulations and best practices

Required

Proficiency in Python and R programming languages

Required

Strong SQL querying and data manipulation skills

Required

Experience with Azure cloud platform

Required

Experience with DevOps, CI / CD pipelines, and version control systems

Required

Working in agile, multicultural environments

Required

Strong troubleshooting and debugging capabilities

Required

Design and develop scalable data pipelines using Apache Spark on Databricks

Required

Optimize Spark jobs for performance and cost-efficiency

Required

Integrate Databricks solutions with cloud services (Azure Data Factory)

Required

Ensure data quality, governance, and security using Unity Catalog or Delta Lake

Required

Deep understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL

Required

Hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake

Preferred

Knowledge of ML libraries (MLflow, Scikit-learn, TensorFlow)

Preferred

Databricks Certified Associate Developer for Apache Spark

Preferred

Azure Data Engineer Associate