Staff Data Infrastructure Engineer

View full description

Requirements:

• 8+ years of experience as a Data Infrastructure Engineer or related roles.
• Deep expertise in provisioning and managing data infrastructure components like Kafka, Spark, and Airflow.
• Extensive experience with cloud services (compute, storage, networking, identity management, etc.).
• Strong experience with containerization technologies (Docker and Kubernetes).
• Advanced knowledge of infrastructure as code tools (Terraform and Pulumi).
• Strong understanding of networking concepts (VPCs, load balancers).
• Extensive experience with monitoring and logging tools.
• Strong problem-solving skills and attention to detail.
• Excellent leadership, communication, and mentoring skills.

Responsibilities:

• Oversee the provisioning and deployment of infrastructure using Pulumi for systems like Kafka, Spark, Airflow, and Athena on AWS.
• Design and implement strategies for scaling Airflow, Kafka, and Spark clusters.
• Lead efforts in optimizing performance and capacity planning with a focus on fault tolerance.
• Manage secure communication between services through VPCs and load balancers.
• Ensure compliance with regulations and manage IAM roles and security patches.
• Architect high-availability solutions and oversee S3 data lake management.
• Implement monitoring and logging tools and develop automation tools for engineers.
• Participate in incident response, interviews, and code reviews while mentoring junior engineers.