Senior Data Engineer

View full description

Requirements:

• Thoughtful, conscientious, and self-directed.
• Experience working with data engineering services on major cloud providers.
• Minimum of 3 years of experience in a data role involving designing and building data stores, feature engineering, and building reliable data pipelines that handle high loads.
• Proven ability to make independent decisions regarding data processing strategy and architecture.
• At least 2 years of professional software development experience in a role other than data engineering.
• Significant experience coding and developing in Python.
• Experience in building and maintaining distributed data pipelines.
• Experience working with Kafka infrastructure and applications.
• Deep understanding of SQL and NoSQL databases (preferably Clickhouse).
• Familiarity with public cloud providers (AWS or Azure).
• Experience with CI/CD and orchestration platforms: Kubernetes, containerization, and microservice design.
• Familiarity with distributed systems and architectures.

Responsibilities:

• Maintain, extend, and improve existing data/ML workflows, and implement new ones to handle high-velocity data.
• Provide interfaces and systems that enable ML engineers and researchers to build datasets on demand.
• Influence data storage and processing strategies.
• Collaborate with the ML team, as well as frontend and backend teams, to build out our data platform.
• Reduce time-to-deployment for dashboards and ML models.
• Establish best practices and develop pipelines that enable ML engineers and researchers to efficiently build and use datasets.
• Work with large datasets under performance constraints comparable to those at the largest companies.
• Iterate quickly, focusing on shipping early and often, ensuring that new products or features can be deployed to millions of users.