Senior Software Engineer, Big Data, Tech Enablement
New
Z
ZipRecruiterData infrastructure
Most US-based positions can also be performed remotelyFull-TimeSenior
Salary190,000 - 250,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years of professional software development experience
- Required Skills
- AWSApache AirflowKafkaKubernetesSparkDistributed Systems
Requirements
- Have 4+ years of professional software development experience, including building and operating data pipelines that feed into a customer-facing product.
- Have hands-on Apache Spark experience writing, tuning, and debugging batch and/or streaming jobs at scale.
- Have experience deploying and operating containerized services and batch jobs.
- Have solid computer science fundamentals in data structures, algorithms, distributed systems, and testing.
- Have working knowledge of AWS, including S3, IAM, EC2, and MSK or equivalents.
- Have experience instrumenting jobs with metrics, logs, and alerts.
- Have experience with a data lake stack, such as Parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake.
- Have streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming.
- Have experience with a workflow orchestrator such as Apache Airflow, including DAG authoring, backfills, dependency management, and on-call for scheduled pipelines.
- Have experience deploying services on Kubernetes with Docker.
- Have a BS, MS, or PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience.
Responsibilities
- Design, build, deploy, and maintain data pipelines and systems that make web events available in the data lake and through event streams.
- Process large, evolving datasets using batch and streaming approaches, balancing speed, efficiency, and reliability.
- Build and maintain frameworks and systems that accelerate ML development and improve its velocity, quality, observability, and reliability.
- Develop frameworks and tooling that improve data documentation and make it easier for people and automated processes to add, maintain, and discover documentation.
- Build and maintain Sift, an agentic data exploration and analysis tool.
View Full Description & ApplyYou'll be redirected to the employer's site
