- Architect and implement large-scale data pipelines for model training and research workflows across text, image, audio, and video datasets.
- Partner with research and engineering teams to curate, clean, and manage datasets for multimodal model pre-training and mid-training.
- Develop tools and strategies for scalable data ingestion, labeling, filtering, augmentation, and storage.
- Manage data quality, reliability, privacy, ethical considerations, and compliance throughout the data lifecycle.
- Optimize data processing, transformation, and delivery for large-scale distributed training pipelines.
- Prototype and productionize methods for dataset creation, management, and continuous improvement.
- Integrate research-driven data advancements into production-ready systems.
- Apply emerging data engineering and ML data management practices to the systems.
AWSPythonSQL+5 more