Staff Data Engineer, Observability Data Lake

View full description

Requirements:

• A minimum of 7 years of experience in database engineering, with a focus on SQL query optimization and data management.
• Strong proficiency in SQL and experience with query optimization techniques.
• Expertise in managing and optimizing both structured and unstructured data.
• Experience with ClickHouse for high-speed data storage and retrieval.
• Familiarity with data pipeline tools, particularly Vector.dev.
• Strong analytical skills, with a focus on delivering actionable insights and improvements.
• Excellent problem-solving skills and the ability to work collaboratively in a dynamic, fast-paced environment.
• Strong communication and interpersonal skills, capable of working effectively with diverse teams.
• Experience with modern development tools and practices, including version control, continuous integration/continuous deployment (CI/CD), and agile methodologies.

Responsibilities:

• Optimize SQL queries to ensure high performance and efficiency in data processing and retrieval.
• Manage and optimize the storage and retrieval of both structured and unstructured data within the observability data lake.
• Collaborate with the Technical Lead and other team members to support the architecture, engineering, and optimization of the observability data lake.
• Leverage ClickHouse for high-speed data storage and retrieval, ensuring seamless integration with data pipelines and visualization tools.
• Work closely with the data pipeline team to ensure efficient data ingestion and correlation at scale using Vector.dev.
• Implement best practices for database management, including indexing, partitioning, and query optimization.
• Create and maintain documentation for database schemas, query optimization techniques, and data management processes.
• Troubleshoot and resolve database-related issues, providing expert guidance and solutions to enhance performance.
• Stay updated on the latest technologies and practices in database management and optimization.
• Participate in project meetings, providing technical oversight and contributing to the successful delivery of observability solutions.