We are seeking a highly skilled and innovative Principal Software Engineer to join our Data Solutions Engineering team, part of a major initiative within the Data Solutions (DS) business unit.
This position plays a critical role in developing a world-class tick data historical database, leveraging Big Data tools like Snowflake to service various client-facing applications.
As part of this role, you will be working closely with Product Subject Matter Experts (SMEs) and collaborating with both internal teams and external vendors to structure, optimize, and enhance data services.
Key responsibilities for this accomplished developer will include but are not limited to the following :
Design and prepare tick data for ingestion into Snowflake, collaborating with internal and external teams to ensure best practices are followed.
Fine-tune Snowflake database performance, managing partition keys and indexes while minimizing costs and optimizing data consumption for applications.
Build data pipelines for applications, including End-of-Day bulk feeds, on-demand APIs, and Snowflake data sharing.
Improve tick data offerings by researching technologies, loading Level 2 data, and optimizing time-binned data access.
Explore, design, and architect potential new storage layers and database architectures that could outperform Snowflake for managing tick history data .
Lead proof of concept (POC) initiatives to evaluate the feasibility, performance, and scalability of these alternative solutions.
Design and architect scalable APIs and data connector frameworks.
Leading the architecture, design, development, and launch of high available, low latency, flexible and scalable APIs.
Mentor and assist other junior engineers in design, implementation, and code reviews.
Stay up to date with advancements in Big Data technologies and data warehousing tools.
Ensure adherence to data security, privacy, and compliance standards.
Drive innovation by evaluating emerging tools and technologies that could improve data management and performance.
Determine operational feasibility by evaluating analysis, problem definition, requirements, solution development and proposed solutions.
Demonstrate a thorough knowledge of data structures and algorithms, object-oriented programming, and software engineering principles.
Ensuring that deployed products are properly maintained throughout their lifecycle.
This position requires to participate in on-call support on a rotation basis with the team.
Minimum Requirements :
10+ years of related work experience with a focus on data engineering and cloud data platforms.
Critical Skills :
Experience in building RESTful APIs, data pipelines, and managing bulk data processes.
Expertise in database architecture, performance tuning, and data ingestion techniques.
Demonstrated expertise in Python and related scripting languages to automate complex data workflows, optimize ETL processes, and develop robust data integration solutions.
Strong analytical skills with the ability to make data-driven decisions.
Experience working with any relational database.
Experience working with Cloud infrastructure (AWS, Azure etc.)
Experience working with Snowflake or related technologies.
Knowledge of financial market data, tick data , and Level 2 data is a big plus.
Familiarity with various levels of testing - unit, regression, integration, and load is a plus.
Additional or Desired Skills :
Experience with finance and financial market data is a plus.
Experience with Big Data technologies (e.g., Spark, Hadoop) is a plus.
Understanding of Java and frameworks like Spring Boot is a plus.
Familiarity building RESTful services is a plus.
Experience with NoSQL storage is a plus.
Experience with AWS tools and technologies is a plus.
Experience with multithreaded, caching, and high availability software development is a plus.
Self-starter with deep interest in learning new technologies and taking different approaches to solving complex problems.
Strong written and interpersonal communication skills to interact with business analysts, peers, and management.
Able to work as part of a geographically diverse team as an individual contributor and within a team as well.
Ability to lead and take full ownership of assigned tasks.
Ability to articulate and quickly adopt development best practices.
Ability to create and review documentation.
Education :
Bachelor’s Degree or equivalent in Computer Science or related field
The budgeted salary range for this position in the states of Connecticut and New York is $160,000.00 - $210,000.00
Principal Software Development Engineer in Test
The Principal Test Engineering role is responsible for ensuring that all applications, system products, projects, and solutions meet defined enterprise quality standards. Responsible for the provisioning of test environments for the team, and assist with coordination to resolve issues. Mitigates ris...
Principal Full Stack Software Engineer
Veson Nautical is a well-established and rapidly growing software company working to provide end-to-end logistical, operational, and analytical solutions to propel the efficiency and effectiveness of Maritime Commerce. Principal Full Stack Software Engineering Opportunity!. The Veson Nautical team i...
Senior Full Stack Software Engineer (Farming & Big Data)
You’ll be instrumental in crafting unique approaches to solving difficult problems, contribute in the continuous improvement of our engineering practices, helping to build out data models, outlining public APIs, and open 3rd-party SDKs. Senior Software Engineer - Greenfield Development (Full-time). ...
Principal Data Engineer
A Principal data engineer will be responsible for driving data integration strategy ensuring the scalability, security, and performance of our data systems and ETL processes to extract, transform, and load data from various sources into our data warehouse. National Grid is hiring a Principal Data En...
Senior Principal Data Engineer
Deep data architecture experience developing reference architectures, data flows and data integration, and data quality solutions designs appropriate for near real-time data needs in both operational and analytic data environments. Under the Interim Director, Enterprise Systems, support the Broad's ...
Principal Software Engineer - (Qt/QML)
Their mission is to improve patient outcomes by developing cutting-edge medical devices and software solutions. They are seeking a highly skilled and experienced Qt Software Engineer to join their team in Boston. Collaborate with cross-functional teams including software developers, hardware enginee...
Principal Software Engineer
SS&C Technologies is an Equal Employment Opportunity employer and does not discriminate against any applicant for employment or employee on the basis of race, color, religious creed, gender, age, marital status, sexual orientation, national origin, disability, veteran status or any other classif...
Principal Software Engineer
Provide engineering troubleshooting assistance to customer support teams and other development teams within Charles River. Why this job is important to us:. CRD helps create enterprise investment management software solutions for large institutions in the areas of institutional investment, wealth ma...
Principal Software Engineer
This position is for an experienced Software Engineer in the Core Vision Technology team, focused on developing the best-in-class computer vision algorithms/AI models that power Cognex's industrial bar code readers and 2D vision tools on a broad range of platforms. Companies around the world rely on...
Principal Software Engineer
We are seeking a Principal Software Engineer to join our multidisciplinary team responsible for developing next generation multi-spectral (infrared, low-light, and visible light) imaging systems. Create and develop a team environment, including building collaborative relationships and participating ...