Software Engineer III, Data Platform
New
This is a fully remote role with the option to work hybrid if a commutable distance from our Salem, OR, Pittsburgh, PA, or Fremont, CA offices.Full-TimeSenior
Salary144,000 - 224,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 3-5 years
- Required Skills
- AWSPythonJavaKafkaData engineeringSparkMicroservicesScala
Requirements
- 3-5 years of experience building and scaling cloud-based data platforms (AWS, GCP, or Azure).
- Proficiency in big data frameworks like Spark, Kafka, Beam, or Flink.
- Familiarity with OSS data formats like Parquet, Arrow, Iceberg and protocols like Avro and Protobuf.
- Experience with observability tools like OpenTelemetry and Prometheus.
- Strong proficiency in Python.
- Proficiency in at least one JVM language (Java or Scala).
- Experience with microservices.
- Proven experience handling diverse data types including unstructured logs/video and structured telemetry.
Responsibilities
- Design and implement scalable data pipelines for ingesting and processing telemetry, logs and sensor data.
- Develop systems to handle complex multimodal logs, ensuring precise synchronization between video, audio, IMU, and joint-state data.
- Contribute to the development of data ingestion agents running on-robot.
- Implement and optimize data formats like MCAP and Parquet for rapid querying.
- Develop and maintain observability systems using fluentbit and OpenTelemetry.
- Use Spark, Kafka, or Flink to transform raw robot data into structured datasets.
- Collaborate with AI and Hardware teams to meet data needs for model training.
- Build and optimize tools for business intelligence using AWS S3, Athena, and Sagemaker.
View Full Description & ApplyYou'll be redirected to the employer's site
