Data Engineer

This international financial services industry leader has an immediate 3+ month contract-to-hire opportunity in Fort Worth, TX for a Data Engineer.

This position requires working onsite in downtown Ft. Worth 2 days per week. Will be responsible for helping the reporting team in building and maintaining reliable and scalable data pipelines.

This position will leverage your expertise in Python, SQL, and Azure cloud technologies to extract, transform, and load data efficiently, enabling seamless data access and analysis for accounting business users.

Requires a proficiency with Databricks coding on MS Azure cloud platform, along with advanced SQL skills. This position involves a high level of coordination with other departments and third-party software vendors.

Contract Duration : 3+ Month Contract-to-Hire.

Required Skills & Experience

• Experience with Databricks platform on Microsoft Azure
• Advanced SQL Skills
• 4-6 years of hands-on data engineering experience
• 3-5 years of hands-on experience with processing large data sets required
• 3-5 years of hands-on experience with SQL, data modeling, relational databases and / or no SQL databases required
• Bachelor’s Degree in related field or equivalent work experience required
• Experience with processing large datasets using Hadoop, Spark, Kafka or similar distributed systems
• Experience with PySpark and distributed data processing frameworks
• Understanding of cloud computing technologies, business drivers and emerging computing trends
• Experience with ingesting various source data formats such as JASON, Parquet, Sequencefile, Cloud databases, MQ, Relational databases such as Oracle
• Working knowledge of Object Storage technologies to include but not limited to Data Lake Storage Gen2, ADLS etc.
• Working knowledge of unity catalog and data governance principles
• Working knowledge of Agile development / SAFe, Scrum and Application Lifecycle Management
• Strong background with source control management systems (GIT or Subversion); Build Systems (Maven, Gradle, Webpack); Code Quality (Sonar);

Artifact Repository Managers (Artifactory), Continuous Integration / Continuous Deployment (Azure DevOps)

• Experience with Python and SQL
• Creating and maintaining ETL processes and data pipelines
• Knowledgeable of best practices in information technology governance and privacy compliance
• Knowledge on auto finance industries core business functions
• Ability to establish priorities and meet tight deadlines.
• Ability to interact collaboratively with internal customers
• Demonstrated quantitative skills and data analytical skills
• Solid design, coding, testing and debugging skills

What You Will Be Doing

• Coding, testing, deploying, orchestrating, monitoring, documenting and implementing data pipelines utilizing the Microsoft Azure Databricks platform.
• Utilizing technologies like Spark, Delta Lake and Unity Catalog for efficient data processing and governance.
• Implementing the Medallion architecture pattern for robust and modular data pipelines.
• Implementing CI / CD pipelines for automated testing and deployment of data pipelines using Azure DevOps.
• Configuring and managing Azure Data Lake Storage for secure and scalable data storage.
• Collaborating and working closely with ABS data analysts, accounting business partners to identify, capture, collect and format data from the external sources, internal systems, and data warehouse to extract features of interest.
• Monitoring and optimizing data pipelines for performance and reliability.
• Staying up-to-date with the latest advancements in Databricks, Azure, and big data technologies.
• Educating and developing ETL developers on data engineering cloud based initiatives so as to enable transition to data engineer and practice.
• Monitoring the production reporting environment on daily and monthly basis.
• Data Analytics to analyze data and make business decisions on the data.
30+ days ago