Azure Databricks Data Engineer -Generative AI & Advanced Lakehouse SolutionsHirekeyz Inc • Atlanta, GA, United States
Azure Databricks Data Engineer -Generative AI & Advanced Lakehouse Solutions
Hirekeyz Inc • Atlanta, GA, United States
job_description.job_card.variable_days_ago
serp_jobs.job_preview.job_type
- serp_jobs.job_card.full_time
- serp_jobs.filters_job_card.quick_apply
job_description.job_card.job_descriptionDesign reusable and version-controlled data transformation workflows using dbt (Data Build Tool) within Databricks. Optimize Spark jobs via adaptive query execution, caching, partitioning, and Z-ordering for performance and cost efficiency. Implement alerts and notification mechanisms (e.g., via Databricks Jobs, Lake Flow, or Azure Monitor) for proactive pipeline monitoring. Package and deploy reusable data artifacts using Databricks Asset Bundles (DABs) to standardize deployments across environments. Develop GenAI-powered assistants and bots leveraging Azure OpenAI, Lang Chain, and vector databases (Azure Cognitive Search, Pinecone, etc.). Integrate Retrieval-Augmented Generation (RAG) pipelines for context-aware enterprise chatbot experiences. Enable conversational analytics and document-based Q&A using company data sources through LLM integrations. Collaborate with ML engineers and solution architects to deploy AI features on Azure Kubernetes Service (AKS) or Azure App Service. Bachelor's or Master's degree in Computer Science, Information Systems, or related field. 8+ years of hands-on experience in data engineering on Azure Databricks. Proficiency in PySpark, SQL, Delta Lake, and Lake Flow. Deep understanding of Unity Catalog & Meta store governance, Spark Streaming & Auto Loader ingestion, ACID Transactions and Delta Lake optimization, Medallion Architecture best practices, SCD1 / SCD2 implementation, Streaming Tables, and SQL Warehouse operations. Experience with dbt (Data Build Tool) for modular data transformations. Strong knowledge of CI / CD pipelines (Azure DevOps, GitHub Actions) and environment management using Databricks Asset Bundles. Familiarity with Generative AI frameworks (Azure OpenAI, Lang Chain) and LLM integration patterns. Experience with Azure Synapse, Power BI, and Azure Data Factory (ADF). Familiarity with Data Governance tools (e.g., Azure Purview). Understanding of MLOps / AIOps and Lakehouse AI convergence patterns. Knowledge of cost optimization and workload tuning in Databricks. Excellent analytical and troubleshooting capabilities. Strong communication and documentation skills. Ability to work cross-functionally with AI, data science, and business teams. Passion for exploring cutting-edge data & AI innovations.
Role : Azure Databricks Data Engineer Generative AI & Advanced Lakehouse Solutions
Location : Atlanta, GA ( On-site)
Duration : 12 Months C2H
Job Summary :
Our client is seeking an innovative Azure Databricks Data Engineer experienced in building scalable data lakehouse architectures and Generative AI (GenAI) solutions. The ideal candidate will design and operationalize advanced data pipelines using Azure Databricks, Unity Catalog, Delta Lake, and Lake Flow, while integrating LLM-based AI assistants and chatbots powered by Azure OpenAI.
Key Responsibilities :
(a) Data Engineering & Lakehouse Development :
- Design, develop, and optimize data pipelines using Azure Databricks, Auto Loader, and Spark Structured Streaming for batch and real-time data processing.
- Implement Delta Lake for unified, reliable, and ACID-compliant data storage.
- Build Lake Flow declarative pipelines for simplified orchestration and dependency management across ingestion, transformation, and serving layers.
- Apply Medallion Architecture (Bronze Silver Gold) principles for modular, reusable data modeling.
- Utilize Unity Catalog and Meta store for centralized governance, lineage, and fine-grained data access control across workspaces.
- Develop and maintain SQL Warehouses for analytics and BI consumption.
- Implement SCD Type 1 & Type 2 (Slowly Changing Dimensions) logic for historical tracking and data consistency.
- Build and maintain Streaming Tables to enable continuous processing and near real-time analytics.
(b) Data Transformation & Optimization :
(c) Generative AI & Intelligent Automation :
Required Skills & Qualifications :
Preferred Skills :
Soft Skills :
serp_jobs.job_alerts.create_a_job
Azure Data Engineer • Atlanta, GA, United States
Job_description.internal_linking.related_jobs
Employment Type : Full-Time, Mid-level.Department : Business Intelligence.CGS is seeking a passionate and driven Data Engineer to support a rapidly growing Data Analytics and Business Intelligence pl...serp_jobs.internal_linking.show_more
CTG is seeking to fill a Data Engineer opening for our client in Atlanta, GA.Design and implement data models for applications and related systems, translating business requirements into logical an...serp_jobs.internal_linking.show_more
Local Candidate only, Alpharetta, GA.We are seeking a highly skilled Data Engineer / Cloud Data Solutions engineer with experience in Google Cloud Platform (GCP) to design, build, and optimize scal...serp_jobs.internal_linking.show_more
At EY, we're all in to shape your future with confidence.We'll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go.Join EY and help ...serp_jobs.internal_linking.show_more
CCI CORPORATE SERVICES LLC Lead Data Engineer positions offered in Atlanta, GA.Exercise responsibility over technology solutions to support the performance, tracking and measurement of paid media a...serp_jobs.internal_linking.show_more
CapTech is an award-winning consulting firm that collaborates with clients to achieve what’s possible through the power of technology.
At CapTech, we’re passionate about the work we do a...serp_jobs.internal_linking.show_more
We are looking for an experienced Data Engineer to join our team in Johns Creek, Georgia, on a Contract to permanent basis.
This position offers an exciting opportunity to design and optimize data p...serp_jobs.internal_linking.show_more
Job title : Director, Data Engineer.Reports to : SVP, Data & Analytics.Location : Alpharetta, GA or Remote.Priority Technology Holdings, Inc.
Our vision is to eliminate the barriers to unlocking re...serp_jobs.internal_linking.show_more
AI / Data Engineer – Core Role at WickedFile.Automotive SaaS, AI-Driven Analytics.WickedFile is on a mission to redefine how automotive repair shops understand and act on their financial and operatio...serp_jobs.internal_linking.show_more CCI CORPORATE SERVICES LLC Lead Data Engineer positions offered in Atlanta, GA.Exercise responsibility over technology solutions to support the performance, tracking and measurement of paid media a...serp_jobs.internal_linking.show_more CCI CORPORATE SERVICES LLC Lead Data Engineer positions offered in Atlanta, GA.Exercise responsibility over technology solutions to support the performance, tracking and measurement of paid media a...serp_jobs.internal_linking.show_more 
Demonstrates strong deductive reasoning and structured problem-solving, breaking down complex and ambiguous challenges into clear hypotheses and logical conclusions.
Balances short-term tactical nee...serp_jobs.internal_linking.show_more CCI CORPORATE SERVICES LLC Lead Data Engineer positions offered in Atlanta, GA.Exercise responsibility over technology solutions to support the performance, tracking and measurement of paid media a...serp_jobs.internal_linking.show_more 
Join a high-performing team and design the data infrastructure that powers decision-making at a leading institutional asset manager.
Do you thrive at the intersection of data, technology, and busine...serp_jobs.internal_linking.show_more
Specifically, you will be working on projects involving leveraging AI / ML for Data Management Efficiencies & Query Optimizations.
Collaborate with cross-functional teams such as Data Scientists, ...serp_jobs.internal_linking.show_more
Heritage Distribution Holdings is building a next-generation business management platform from the ground up — and we’re looking for a hands-on Senior Data Engineer to deliver and scale...serp_jobs.internal_linking.show_more
Atlanta, Georgia (Hybrid Remote- 2 days onsite / 3 days remote).The Data Engineer is a vital member of the application services team responsible for designing, building, and maintaining the infrastr...serp_jobs.internal_linking.show_more CCI CORPORATE SERVICES LLC Lead Data Engineer positions offered in Atlanta, GA.Exercise responsibility over technology solutions to support the performance, tracking and measurement of paid media a...serp_jobs.internal_linking.show_more CapTech is an award-winning consulting firm that collaborates with clients to achieve what’s possible through the power of technology.
At CapTech, we’re passionate about the work we do a...serp_jobs.internal_linking.show_more CCI CORPORATE SERVICES LLC Lead Data Engineer positions offered in Atlanta, GA.Exercise responsibility over technology solutions to support the performance, tracking and measurement of paid media a...serp_jobs.internal_linking.show_more
Data Engineer
Contact Government Services, LLC • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Data Engineer
Computer Task Group, Inc • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
GCP Data Engineer
Cedent Consulting • Alpharetta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Data Engineer - Snowflake SME
Ernst & Young Oman • Alpharetta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Lead Data Engineer
Cox Communications • Vinings, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Lead Data Engineer (AWS, Azure, GCP)
CapTech Consulting • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Data Engineer
Robert Half • Duluth, GA, US
serp_jobs.job_card.permanent
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Director, Data Engineer
Priority Technology Holdings, LLC • Alpharetta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
AI / Data Engineer
WickedFile • Alpharetta, Georgia, United States
serp_jobs.job_card.full_time
serp_jobs.filters_job_card.quick_apply
serp_jobs.last_updated.last_updated_30
Lead Data Engineer
Cox Communications • Lake City, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Lead Data Engineer
Cox Communications • Marietta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Data Engineer
Deploy • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Lead Data Engineer
Cox Communications • Brookhaven, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Data Analytics Engineer
EARNEST Partners • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Big Data engineer
Cpctechno • Kennesaw, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Senior Data Engineer
Heritage Distribution • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_1_day • serp_jobs.job_card.promoted
Senior Data Engineer
TIER4 GROUP • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Lead Data Engineer
Cox Communications • Scottdale, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Data Engineer (AWS, Azure, GCP)
CapTech Consulting • Atlanta, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Lead Data Engineer
Cox Communications • Powder Springs, GA, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted