Staff Software Engineer, Data Incrementalization

View full description

Requirements:

• Bachelor's degree (or equivalent) in Computer Science or a related field.
• 5+ years of experience in building distributed system architecture, with a particular focus on incremental updates from inception to production.
• Strong programming skills in Python and SQL.
• Deep technical expertise in advanced data structures and algorithms for incremental updating of data stores (e.g., Graphs, Trees, Hash Maps).
• Comprehensive knowledge across all facets of data engineering, including implementing and managing incremental updates in data stores like BigQuery, Snowflake, RedShift, Athena, Hive, and Postgres.
• Orchestrating data pipelines and workflows focused on incremental processing using tools such as Airflow, DBT, Luigi, Azkaban, and Storm.
• Developing and optimizing data processing technologies and streaming workflows for incremental updates (e.g., Spark, Kafka, Flink).
• Deploying and monitoring scalable, incremental update systems in public cloud environments (e.g., Docker, Terraform, Kubernetes, Datadog).
• Expertise in loading, querying, and transforming large datasets with a focus on efficiency and incremental growth.

Responsibilities:

• Design and build our Cloud Data Warehouse with a focus on incremental updates to improve cost efficiency and scalability.
• Research innovative methods to incrementally optimize data processing, storage, and retrieval to support efficient data analytics and insights.
• Develop and maintain ETL pipelines that transform and incrementally process petabytes of structured and unstructured data to enable data-driven decision-making.
• Collaborate with cross-functional teams to design and implement new data models and tools focused on accelerating innovation through incremental updates.
• Continuously monitor and optimize the Data Platform's performance, focusing on enhancing cost efficiency, scalability, and reliability.