Senior Data Engineer, Databricks on AWS, AVP - Onsite

Who we are looking for

As a Sr. Data Engineering specializing in modern data management, you will be at the forefront of our efforts to transition our data ecosystem to a more efficient and modern platform.

Your expertise in data engineering, AWS cloud technologies, and migration strategies will be crucial in ensuring a seamless migration process.

Collaborating closely with cross-functional teams, you'll design, implement, and optimize data pipelines, enabling the secure and accurate transfer of data to our new AWS-based data warehouse environment.

ONSITE : Due to the role requirements this job needs to be performed primarily in the Boston office with some flex work opportunities available.

What you will be responsible for

• Design, develop, and maintain scalable data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.
• Implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing.
• Utilize Parquet and Avro-formatted data files for efficient storage and retrieval, ensuring data schema compatibility and evolution.
• Leverage Databricks platform on AWS to build and manage data processing workflows and analytics, while adhering to development lifecycle standards.
• Harness the power of Databricks Delta Lake and Parquet files for data warehousing, query optimization, and data versioning.
• Collaborate closely with data analysts and scientists to understand their requirements and provide reliable and timely data solutions.
• Implement robust testing methodologies, including unit testing, integration testing, and end-to-end testing, utilizing Python packages such as pytest.
• Contribute to the pyspark / Python ecosystem by creating reusable components, maintaining internal PyPI packages, and exploring other common Python packages.
• Monitor data pipelines, identify and resolve issues, and ensure data integrity and quality.
• Stay up-to-date with the latest trends and technologies in data engineering, software development, and testing practices, and actively share knowledge with the team.

What we value

These qualifications will help you succeed in this role

• Master’s degree in computer science or a related field.
• Minimum 8-10 years of real world Data Engineering experience working on large scale data projects.
• Strong proficiency in pySpark, Python and shell scripting, with a focus on software engineering best practices and a deep understanding of development lifecycle.
• Experience working with workflow management tools such as Airflow
• Experience with stream processing technologies, preferably Kafka.
• Familiarity with Avro data serialization format and its usage in data engineering workflows.
• Expertise in using Databricks platform on AWS for data processing and analytics.
• Solid understanding of data warehousing concepts and experience with Delta Lake and Parquet files.
• Proficiency in SQL and experience with relational databases.
• Strong testing skills, with experience in implementing and executing unit tests, integration tests, and end-to-end tests using Python packages such as pytest.
• Familiarity with the Python ecosystem, including PyPI packages and their integration into data engineering workflows.
• Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.
• Strong communication skills and ability to effectively communicate complex technical concepts to non-technical stakeholders.

Must have qualifications :

• Working experience with Databricks and pyspark
• Proficiency in writing complex SQLs
• Working experience with cloud platforms like AWS or Azure (preferably AWS)
• Working Experience with Airflow
• Experience working with very large datasets

Nice to have qualifications :

• Experience working with reporting tools such as Tableau
• Past experience working on Machine Learning projects
• Past experience working in finance

If you're an experienced Data Engineering with a track record of successful large scale Data projects and want to play a key role in shaping the future of our data ecosystem, we invite you to apply.

Please submit your resume and a cover letter outlining your relevant experience and how it aligns with the requirements of this role.

Are you the right candidate? Yes!

We truly believe in the power that comes from the diverse backgrounds and experiences our employees bring with them. Although each vacancy details what we are looking for, we don’t necessarily need you to fulfil all of them when applying.

If you like change and innovation, seek to see the bigger picture, make data driven decisions and are a good team player, you could be a great fit.

Why this role is important to us

Our technology function, Global Technology Services (GTS), is vital to State Street and is the key enabler for our business to deliver data and insights to our clients.

We’re driving the company’s digital transformation and expanding business capabilities using industry best practices and advanced technologies such as cloud, artificial intelligence and robotics process automation.

We offer a collaborative environment where technology skills and innovation are valued in a global organization. We’re looking for top technical talent to join our team and deliver creative technology solutions that help us become an end-to-end, next-generation financial services company.

Join us if you want to grow your technical skills, solve real problems and make your mark on our industry.

About State Street

What we do. State Street is one of the largest custodian banks, asset managers and asset intelligence companies in the world.

From technology to product innovation, we’re making our mark on the financial services industry. For more than two centuries, we’ve been helping our clients safeguard and steward the investments of millions of people.

We provide investment servicing, data & analytics, investment research & trading and investment management to institutional clients.

Work, Live and Grow. We make all efforts to create a great work environment. Our benefits packages are competitive and comprehensive.

Details vary by location, but you may expect generous medical care, insurance and savings plans, among other perks. You’ll have access to flexible Work Programs to help you match your needs.

And our wealth of development programs and educational support will help you reach your full potential.

Inclusion, Diversity and Social Responsibility. We truly believe our employees’ diverse backgrounds, experiences and perspectives are a powerful contributor to creating an inclusive environment where everyone can thrive and reach their maximum potential while adding value to both our organization and our clients.

We warmly welcome candidates of diverse origin, background, ability, age, sexual orientation, gender identity and personality.

Another fundamental value at State Street is active engagement with our communities around the world, both as a partner and a leader.

You will have tools to help balance your professional and personal life, paid volunteer days, matching gift programs and access to employee networks that help you stay connected to what matters to you.

Salary Range :

$100,000 - $160,000 Annual

The range quoted above applies to the role in the primary location specified. If the candidate would ultimately work outside of the primary location above, the applicable range could differ.