Senior/Lead DataOps Engineer

View full description

Requirements:

• Strong background in software engineering with proficiency in Python.
• Experience in designing, building, and maintaining data processing pipelines, including data preparation and transformation.
• Hands-on experience with message queues such as RabbitMQ, NATS, gRPC, REST or others.
• Familiarity with cloud infrastructure, ops, and containerized tools like Kubernetes (K8s), Docker or others.
• Familiarity with modern software development practices, such as automated testing, code reviews, and CI/CD pipelines.
• Analytical and problem-solving skills with the ability to troubleshoot complex systems and implement effective solutions.
• Professional or personal interest in Research/Machine Learning/Deep-Learning.
• Great communication skills that enable collaboration with other engineering teams.
• Fluency in English and the ability to articulate complex technical concepts clearly.

Responsibilities:

• Developing robust data pipelines and tools for efficient data processing and preparation to support machine learning engineers (MLEs).
• Writing production code to implement algorithms and data transformation rules.
• Collaborating closely with MLEs, MLOps, and Platform engineers to deploy production-ready models.
• Improving the observability and testing frameworks for production data pipelines and deployed models.
• Writing automated tests for data pipelines to ensure reliability and maintainability.
• Participating in simple exploratory work and experimentation, including tasks such as prompt engineering for GenAI tools and basic ML model experimentation.
• Implementing efficient and scalable data processing workflow and tools to enable researchers.
• Enhancing the functionality of mapping tools to increase the capacity and efficiency of the ML team.
• Documenting workflows, processes, and tools to foster team knowledge sharing.
• Mentoring junior engineers and contributing to team growth through onboarding and collaboration.