Big Data Engineer

Job Description

Compensation :

$55 / hr to $75 / hr. - Exact compensation may vary based on several factors, including skills, experience, and education.

Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching.

Employees in this role are also entitled to paid sick leave and / or other paid time off as provided by applicable law.

A Fortune-300 client of Insight Global in the transportation industry is looking for an experience Big Data Engineer to join their smart, highly skilled team with a passion for technology, where they will work on their state-of-the-art Big Data Platforms.

They must be a very good communicator, both written and verbal, and have some experience working with business areas to translate their business data needs and data questions into project requirements.

The candidate will participate in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions, and perform data ingestion development and support duties.

Day to day expectations include a daily stand up meeting, occasional meetings with the business to collect requirements, and production support on an on-call rotation.

75% of their days will be spent completing whatever task the candidate is working on in development. They will be working alongside 6 other Data Engineers to migrate all use cases to AWS, including data platform and Teradata.

Development is currently in progress but they are close to the QA / production phase. The team culture is easy-going and largely independent, but they pride themselves on their willingness to jump in and help each other whenever needed.

Accountability is another cornerstone of their team.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day.

We are an equal opportunity / affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances.

If you need assistance and / or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to .

To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy : .

Required Skills & Experience

• Hands-on experience with Databricks
• Hands-on experience with high-velocity high-volume stream processing : Apache Kafka and Spark Streaming

o a. Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka

o b. Deep knowledge of troubleshooting and tuning Spark applications

• Hands-on experience with Python ( PySpark / Scala-Spark)
• Experience with Cloud platforms AWS, GCP, etc.

Strong SQL skills with ability to write intermediate complexity queries

Bachelors Degree (preferably in Information Systems, Computer Science, Electrical Engineering, Computer Information Systems, or related field)

Nice to Have Skills & Experience

• Hands on experience with Scala
• Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
• Experience with data ingestion of different file formats across like JSON, XML, CSV
• Knowledge of Unix / Linux platform and shell scripting
• Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
• Strong understanding of Relational & Dimensional modeling
• Experience with GIT code versioning software
• Experience with REST API and Web Services
• Good business analyst and requirements gathering / writing skills
• Experience with database solutions like Kudu / Impala, or Delta Lake or Snowflake or BigQuery

Experience with Traditional ETL tools and Data Modeling

Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching.

Employees in this role are also entitled to paid sick leave and / or other paid time off as provided by applicable law.