- Design, build, and optimize scalable pipelines for ingesting, transforming, and integrating market data, news feeds, and unstructured data.
- Maintain data quality, consistency, and real-time monitoring using data validation tools.
- Develop processes to normalize and organize the data warehouse for use across departments.
- Apply data management practices that support scalable, available, and efficient data storage.
- Ensure infrastructure supports trading and research needs while maintaining data integrity, security, and performance.
- Collaborate with research and analytics teams to understand their data needs and build frameworks for data exploration, analysis, and model development.
- Create tools for overlaying data from multiple sources.
- Optimize hardware and software resources to balance performance with cost control.
- Evaluate and adopt suitable technologies and data management practices.
PythonSQLApache Airflow+6 more