Talent.com
Platform Engineer- Kafka with Linux
Platform Engineer- Kafka with LinuxVDart Inc • New York City, NY, United States
Platform Engineer- Kafka with Linux

Platform Engineer- Kafka with Linux

VDart Inc • New York City, NY, United States
job_description.job_card.1_day_ago
serp_jobs.job_preview.job_type
  • serp_jobs.job_card.temporary
  • serp_jobs.filters_job_card.quick_apply
job_description.job_card.job_description

Role : Platform Engineer- Kafka with Linux.

Location : New York City, NY (Onsite).

Duration : 12+ Months Contract.

Job Description :

  • Core Platform Engineer with deep expertise in Apache Kafka and Linux systems engineering.
  • The ideal candidate will be responsible for designing, deploying, maintaining, and scaling Kafka clusters in mission-critical environments, while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.

Responsibilities :

  • Design, implement, and maintain highly available Kafka clusters in production environments.
  • Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
  • Administer security protocols and access controls to protect the platform.
  • Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)
  • Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.
  • Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
  • Assist with planning and managing cluster resources to support future growth.
  • Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.
  • Manage a tune Kafka Connect, Kafka Streams, and MirrorMaker as needed for data pipelines.
  • Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).
  • Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform).
  • Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
  • Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.
  • Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
  • Perform root cause analysis and resolution of platform-level incidents and outages.
  • Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.

    • Minimum Qualifications :

  • BE / B Tech / MCA.
  • Excellent written and verbal communication skills.
  • Min 5+ years of experience.
  • Experience in Finance Domain.

    • Preferred Qualifications / Skills :

  • Strong hands-on experience with Apache Kafka (2.x or 3.x), including cluster administration and tuning.
  • In-depth experience with Linux system internals, kernel tuning, filesystems, and performance troubleshooting.
  • Experience in managing Kafka brokers in production environments with millions of messages per second.
  • Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
  • Scripting and automation skills in Shell, Python, or Go.
  • Experience in container orchestration platforms like Docker and Kubernetes.
  • Familiarity with SSL / TLS, SASL, RBAC, and encryption in-transit / at-rest for Kafka.
  • Expertise in monitoring and alerting for Kafka and Linux systems.
  • Strong understanding of distributed systems, message queues, and event-driven architectures.
    • serp_jobs.job_alerts.create_a_job

    Linux Engineer • New York City, NY, United States