AI Pipeline Engineer
New
B
Bright Vision TechnologiesAI data infrastructure
100% Remote (U.S.)Full-TimeMiddle
Salary100,000 - 150,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years
- Required Skills
- PythonSparkCI/CDData modelingDistributed Systems
Requirements
- Hold a Bachelor’s or Master’s degree in Computer Science or a related field.
- Have six or more years of data engineering experience, with significant work supporting ML or AI workloads.
- Demonstrate strong proficiency in Python and at least one JVM or systems language.
- Have deep experience with modern data processing frameworks such as Spark, Ray, or Beam.
- Have hands-on experience operating petabyte-scale storage and pipeline systems.
- Understand distributed systems, data modeling, and storage formats.
- Have experience with dataset versioning, lineage, and reproducibility for ML workflows.
- Be familiar with high-throughput data loading for accelerator-based training.
- Apply software engineering practices including testing, CI/CD, and code review.
- Preferred: experience with large-scale multimodal datasets, data quality tooling, dataset evaluation methodology, privacy-preserving systems, regulated data handling, open-source data infrastructure, or frontier model training pipelines.
Responsibilities
- Design and operate large-scale data pipelines supporting AI training, evaluation, and continual improvement workflows.
- Build ingestion systems for text, image, audio, video, and structured signals.
- Implement data cleaning, deduplication, filtering, and quality assurance at petabyte scale.
- Develop dataset versioning, lineage, and provenance tracking for reproducible training.
- Build high-throughput data loading systems to maximize GPU utilization during training.
- Implement labeling workflows, active learning pipelines, and human-in-the-loop data improvement systems.
- Design storage architectures balancing cost, throughput, and latency across data tiers.
- Build evaluation dataset construction pipelines with integrity and contamination controls.
- Implement data privacy, redaction, and consent enforcement throughout the pipeline.
- Collaborate with ML researchers and engineers and drive observability of data quality, drift, and pipeline health.
View Full Description & ApplyYou'll be redirected to the employer's site
