Big Data Engineer with Hive, Spark, and Scala - Contract W2

Job Description

Job Description

Big Data with Hive, Spark, Scala

Phoenix, AZ

Onsite(Hybrid)

Contract W2

24+ Months

07+ Years

Must-Have :

Minimum 7 years of experience as a Big Data Engineer or Scala Developer, with a strong focus on big data technologies.

Proficient in Scala programming language, including experience with functional programming concepts, type systems, and concurrency.

Hands-on experience with big data technologies, such as Apache Hadoop, Apache Spark, Apache Kafka, and Apache Hive.

Knowledge of data processing patterns and architectural patterns used in big data systems, such as batch processing, stream processing, and lambda architecture.

Familiarity with data modeling techniques and experience in designing efficient data storage solutions, including relational databases, NoSQL databases, and data lakes.

Strong analytical and problem-solving skills, with the ability to identify and address performance bottlenecks and data quality issues.

Excellent communication and collaboration skills, with the ability to work cross-functionally and translate technical concepts to non-technical stakeholders.

Experience with cloud-based big data platforms, such as Amazon EMR, Google Cloud Dataproc, or Microsoft Azure HDInsight.

Familiarity with data visualization and business intelligence tools, such as Tableau, Power BI, or Looker.

Knowledge of machine learning and data science techniques, and experience in integrating them into big data pipelines.

Exposure to DevOps practices and tools, such as Docker, Kubernetes, or CI / CD pipelines.

Company Overview :

We are a leading data-driven organization that leverages the power of big data and advanced analytics to drive business insights and innovation.

We are seeking a talented Big Data Engineer with expertise in Hive, Spark, and Scala to join our team and help us unlock the full potential of our data assets.

Position : Big Data Engineer with Hive, Spark, and Scala

Responsibilities :

Design and implement scalable and efficient big data processing pipelines using Scala and big data technologies like Apache Spark and Apache Hive.

Develop and maintain batch and real-time data ingestion, transformation, and processing workflows.

Integrate and optimize big data technologies, such as Apache Hadoop, Apache Spark, Apache Kafka, and Apache Hive, to create a robust and high-performing data ecosystem.

Write efficient and optimized Scala code for data processing and transformation tasks using Spark's DataFrame and Dataset APIs.

Develop and maintain complex SQL queries and data transformations using Apache Hive.

Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.

Ensure data quality, consistency, and security throughout the data lifecycle, implementing best practices for data governance and compliance.

Optimize the performance and scalability of big data systems, leveraging techniques like partitioning, indexing, and caching.

Automate and streamline data processing tasks using tools like Apache Airflow, Jenkins, or custom scripts.

Stay up-to-date with the latest trends and advancements in the big data and Scala technology landscape, and recommend improvements to the company's data architecture.

Tekdoors Inc. is a leading staffing and IT consulting firm with a global presence and 16+ years of IT consulting experience.

Headquartered in Arizona, we are rated the Top 10 most emerging IT consulting companies in 2024. Our mission is to provide top-quality IT and talent solutions to businesses of all sizes, helping them achieve their goals and stay ahead of the competition.

With offices and operations worldwide, we have the expertise and resources to deliver customized solutions that meet the unique needs of our clients.

Our team of experienced recruiters and consultants has a deep understanding of the IT industry and its ever-changing demands.