Software Engineer, Big Data, Tech Enablement
New
Z
ZipRecruiterData infrastructure
Location: Remote. Most US-based positions can also be performed remotely.Full-TimeMiddle
Salary135,000 - 185,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 2+ years of professional software development experience; preferred: 4+ years with a focus on data-intensive systems.
- Required Skills
- AWSApache AirflowKubernetesApache KafkaSparkDistributed Systems
Requirements
- Have 2+ years of professional software development experience.
- Have experience building and operating data pipelines that feed into a customer-facing product.
- Have hands-on Apache Spark experience writing, tuning, and debugging batch and/or streaming jobs at scale.
- Have experience deploying and operating containerized services and batch jobs.
- Have solid computer science fundamentals, including data structures, algorithms, distributed systems, and testing.
- Preferred: Have 4+ years of professional software development experience focused on data-intensive systems.
- Preferred: Have working knowledge of AWS, including S3, IAM, EC2, and MSK or equivalents, and experience instrumenting jobs with metrics, logs, and alerts.
- Preferred: Have experience with data lake stacks, including Parquet on object storage, data catalogs such as AWS Glue, and open table formats such as Apache Iceberg or Delta Lake.
- Preferred: Have streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.
- Preferred: Have experience with Apache Airflow or another workflow orchestrator, including DAG authoring, backfills, dependency management, and on-call for scheduled pipelines.
- Preferred: Have experience with open table format internals, including schema evolution, partitioning, and compaction.
- Preferred: Have experience deploying services on Kubernetes with Docker.
- Preferred: Have a BS, MS, or PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.
Responsibilities
- Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and through event streams.
- Process large, evolving datasets using batch and streaming techniques.
- Design and maintain frameworks and systems that accelerate machine learning development across teams and products.
- Improve the comprehensiveness and quality of data documentation through processes and tooling.
- Build and maintain Sift, an agentic data exploration and analysis tool.
View Full Description & ApplyYou'll be redirected to the employer's site
