Role : Lead Databricks Engineer with Pyspark & Python Exp
Take the next step in your career now, scroll down to read the full role description and make your application.
Location : NYC, NY OR Iselin, NJ (Hybrid 3 days' work from office)
Duration : 6+ Months
- Query / Redshift experience is good to have
- 3+ years Python (NumPy, Pandas) coding experience
- Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have
- Experience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling
- Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills
- Capable of discussing enterprise-level services independent of technology stack
Job Description
Major Responsibilities
- Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
- Design, develop, and deploy Databricks jobs to process and analyze large volumes of data
- Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines
- Optimize Databricks jobs for performance and scalability to handle big data workloads
- Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks
- Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions
- Demonstrated proficiency with Data Analytics, Data Insights
- Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process
- Azure Synapse / Bigquery / Redshift is good to have
- Perform code reviews to ensure fit to requirements, optimal execution patterns, and adherence to established standards
Skills
- 5+ years' Strong experience in Finance / Banking industry Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
- 10+ years - Enterprise Data Management
- 10+ years - SQL Server-based development of large datasets
- 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding
- Azure Synapse / BigqExperience with Cloud-based data architectures, messaging, and analytics
- Superior communication skills
- Cloud certification(s)
- Any experience with Regulatory Reporting is a Plus
Education
- Minimally a BA degree within an engineering and / or computer science discipline
- Master's degree strongly preferred
J-18808-Ljbffr
Lead Machine Learning Engineer (Python, Spark, MLOPS, AWS)
All qualified applicants will receive consideration for employment without regard to sex (including pregnancy, childbirth or related medical conditions), race, color, age, national origin, religion, disability, genetic information, marital status, sexual orientation, gender identity, gender reassign...
Lead Software Engineer, Python
Ave (22114), United States of America, New York, New YorkLead Software Engineer, PythonDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be ...
Remote Structural Engineer with US Experience
We are currently seeking experienced Structural Engineers with US remote work experience to join our exclusive network. We are looking for highly skilled Structural Engineers with at least two years of remote experience working for a US company. As a part of our exclusive network, you will have the ...
Lead Machine Learning Engineer (Python, Spark, MLOPS, AWS)
Basic Qualifications:Bachelor’s degree At least 6 years of experience designing and building data-intensive solutions using distributed computing (Internship experience does not apply)At least 4 years of experience programming with Python, Scala, or JavaAt least 2 years of experience building, scali...
Lead Machine Learning Engineer ( Python, Spark, MLOPS, AWS)
West 19th Street (22008), United States of America, New York, New YorkLead Machine Learning Engineer ( Python, Spark, MLOPS, AWS). At least 6 years of experience designing and building data-intensive solutions using distributed computing (Internship experience does not apply). At least 4 years of ex...
Lead Software Engineer, DevOps (Python, Kubernetes, Jenkins)
West Creek 4 (12074), United States of America, Richmond, VirginiaLead Software Engineer, DevOps (Python, Kubernetes, Jenkins)Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery ...
Lead Software Engineer - Python
Operate on or build solution required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, AWS and Spark technologies * Leads evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of ...
Lead Software Engineer, Full Stack (Python,AWS,GO)
Center 2 (19050), United States of America, McLean, VirginiaLead Software Engineer, Full Stack (Python,AWS,GO). Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? A...
Lead Python Engineer
Project management experience with an ability to run multiple concurrent projects. Advanced degree in financial engineering or related field is a plus. A Bachelor's Degree in Computer Science, Mathematics, Engineering, or related field. Experience developing risk systems for fixed income and/or fx a...
Lead Software Engineer, Full Stack (Python, TypeScript, Go)
West 19th Street (22008), United States of America, New York, New YorkLead Software Engineer, Full Stack (Python, TypeScript, Go). Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative deli...