Senior Software Engineer, Data Acquisition

New

The permanent ability to work wherever and however you wantFull-TimeSenior

Salary160,000 - 200,000 USD per year

Apply NowOpens the employer's application page

Job Details

Experience: 7+ years of professional experience building or operating backend or infrastructure systems at scale
Required Skills: PythonETLKafkaGoRustDistributed Systems

Have 7+ years of professional experience building or operating backend or infrastructure systems at scale.
Have solid programming experience in Python, Go, Rust, or a similar language.
Have experience with async/await, coroutines, or concurrency frameworks.
Understand software architecture and backend fundamentals, including concurrency, scalability, and fault tolerance.
Understand browser rendering and web application architecture, including authentication, cookies, and HTTP requests and responses.
Have familiarity with network architecture and debugging, including HTTP, DNS, proxies, and packet capture and analysis.
Understand distributed systems concepts, including parallelism, asynchronous programming, backpressure, and message-driven design.
Have experience designing or maintaining resilient data ingestion, API integration, or ETL systems.
Be proficient with Linux/Unix command-line tools and system resource management.
Have familiarity with message queues, orchestration, and distributed task systems such as Kafka, SQS, or Airflow.
Have experience evaluating and monitoring data quality, including consistency, completeness, and reliability across releases.
Be able to work independently and collaborate asynchronously, communicate clearly in writing, and maintain technical design documents.

Contribute to the architecture and improvement of the data acquisition and processing platform.
Use and develop web crawling technologies to capture and catalog data on the internet.
Build, operate, and evolve large-scale distributed systems that collect, process, and deliver data from across the web.
Design and develop backend services for distributed job orchestration, data pipelines, and large-scale asynchronous workloads.
Structure and model captured data to ensure quality and consistency across datasets.
Improve the speed, scalability, and fault tolerance of ingestion systems.
Partner with data product and engineering teams to design new data products and improve existing products.
Write and maintain technical design documents, including pipeline designs, schema designs, and data flow diagrams.
Scope complex projects into milestones and communicate progress, risks, and blockers.

View Full Description & ApplyYou'll be redirected to the employer's site