Sr Software Engineer, Data Platform

View full description

Requirements:

• 10+ years of experience building data platforms/data backends.
• 5+ years of experience working with Python.
• Experience building and optimizing ‘big data’ data pipelines using Spark.
• Familiar with best practices in building CVML pipelines.
• Strong analytic skills related to working with unstructured datasets.
• Working knowledge of message queuing, stream processing, and scalable ‘big data’ stores.
• Experience with relational SQL and NoSQL databases, including MongoDB.
• Experience with data pipeline and orchestration platforms such as Airflow.
• Experience with AWS cloud services like EC2 and RDS.
• Experience with Terraform.
• Strong problem-solving and troubleshooting skills.
• Excellent communication skills.

Responsibilities:

• Create and maintain optimal data platforms for ingesting machine logs, image data, and various types of datasets.
• Assemble large, complex data sets that meet functional and non-functional business requirements.
• Identify, design, and implement internal process improvements.
• Build infrastructure for data extraction, transformation, and loading using Python, SQL, Databricks Spark, and AWS.
• Build and enhance CVML pipelines, integrate data within Kubeflow and Databricks.
• Assist stakeholders with data-related technical issues.
• Create data tools for analytics and optimize workflows.
• Triage and fix support issues related to data anomalies.
• Collaborate with cross-functional teams to deliver data solutions.
• Maintain technical documentation.