Pipeline Engineer
New
C
CloudLinuxLinux Infrastructure Security
Work from any location worldwideFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- English
- Experience
- 5+ years
- Required Skills
- PythonClickhouseGoGrafanaPrometheusRustCI/CD
Requirements
- 5+ years of professional backend, platform, or infrastructure engineering experience.
- Demonstrable experience building and operating multi-stage data or automation pipelines (CI/CD, ETL, job orchestration).
- Deep proficiency in at least one of: Python, Go, or Rust.
- Strong systems design judgment with a focus on reliability, scalability, and maintainability.
- Practical experience with workflow orchestration and job scheduling tools (e.g., Airflow, Temporal, Prefect).
- Deep understanding of reliability engineering concepts like idempotency, retries, and graceful degradation.
- Hands-on experience with observability tools (e.g., Prometheus, Grafana, LGTM stack).
- Deep CI/CD experience, ideally with GitLab CI, Docker, and container-based testing.
- Experience with object storage (S3/Ceph) and large-scale analytical databases (e.g., ClickHouse).
- Excellent debugging skills across system, network, and data layers.
- Strong communication skills and experience working in a distributed team.
- Proficiency in spoken and written English.
Responsibilities
- Design, build and operate end-to-end automated pipelines for threat intelligence and security rule deployment.
- Transform multi-stage batch jobs into resumable, idempotent, and observable systems with explicit state machines.
- Define and enforce latency budgets and SLOs per stage with actionable monitoring.
- Build observability layers including metrics, dashboards, alerting, and health gates.
- Design and implement automated guardrails including rollbacks, blast-radius limits, and kill switches.
- Maintain systems by eliminating manual steps and reducing operational complexity.
- Write unit and integration tests for complex concurrency and multi-stage state logic.
- Investigate and resolve complex technical issues across ClickHouse, GitLab CI, S3, and Prometheus/Grafana.
- Collaborate with security analysts and the Server team on system architecture.
View Full Description & ApplyYou'll be redirected to the employer's site
