Data Systems Engineer, Rust

View full description

Requirements:

• Experience in database system internals.
• Familiarity with Rust is highly desirable.
• Knowledge of Apache DataFusion and its ecosystem is highly desirable.
• General knowledge of distributed data systems and resource management is also highly desirable.
• A background in data ingestion from blockchains or other high-throughput sources is a plus.
• Adaptability to a dynamic and fully-remote work environment is required.
• A rigorous approach to software development is essential.

Responsibilities:

• Leveraging and contributing to Apache DataFusion, monitoring upstream developments and learning its codebase.
• Implementing storage strategies for blockchain and web3 data sources, considering industry standard formats.
• Developing the internals of a streaming processing system, applying incremental techniques for blockchain updates.
• Applying resource isolation for CPU, RAM, and storage in multi-tenant environments.
• Building data transformation pipelines for blockchain data with operations like aggregations and joins.
• Optimizing compaction and garbage collection in high-throughput database systems.
• Creating supportive tooling including tracing and profiling.