- Design and implement new integrations with data products, requiring deep knowledge about APIs, data models, and behavior of each data source.
- Make necessary changes to architecture and implementation to scale the data ingestion engine for large instances.
- Add support for completely new integration types, defining how they will be displayed and integrated within the Sifflet application.
- Lead technical improvements to the codebase and architecture, addressing challenges related to modularization, testability, and reliability when integrating with many external services.
- Help improve team standards and processes, both around technical decisions and product design.
- Build a new model for lineage capabilities, merging data from various sources into an easy-to-query model.
- Optimize queries issued by the ingestion engine to reduce customer costs.
- Fetch query history from all sources and use it as input for automated root cause analysis.
PythonAWS EKSJava+12 more