Staff Software Engineer, Tech Foundations
New
J
JobgetherObservability infrastructure
Based in the United StatesFull-TimeStaff
SalaryCompetitive annual base salary ranging from $212,000 to $265,000 USD. Potential eligibility for performance-based bonuses. Equity compensation opportunities.
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 9 years of software engineering experience
- Required Skills
- AWSKubernetesGoPrometheusDistributed Systems
Requirements
- Have at least 9 years of software engineering experience.
- Have a proven track record designing, building, and operating large-scale distributed systems in production.
- Have led the technical direction of complex systems or multi-team initiatives, including architecture, stakeholder alignment, delivery, and production operations.
- Bring deep experience at scale in at least one telemetry domain: metrics, logging, or distributed tracing.
- Have strong software development capabilities in Go or Java/Kotlin.
- Be able to navigate unfamiliar codebases and contribute to upstream open-source projects.
- Have hands-on experience with cloud-native technologies, particularly Kubernetes and AWS.
- Have experience with service mesh solutions such as Istio or Envoy.
- Be able to produce technical design documents, explain architectural tradeoffs, and present investment recommendations to engineering leadership.
- Preferred: contribute to OpenTelemetry, Prometheus, Grafana Mimir, VictoriaMetrics, OpenSearch, Jaeger, or comparable projects.
- Preferred: familiarity with time-series storage internals, high-cardinality datasets, query optimization, anomaly detection, automated correlation, or SLO-based alerting.
- Preferred: experience with telemetry cost attribution, infrastructure cost optimization, AI-agent-facing APIs or tools including MCP servers, or columnar analytics platforms such as ClickHouse.
Responsibilities
- Define the technical vision and architecture for major areas of the Observability platform, from design and prototyping through production deployment and operations.
- Design, build, and maintain scalable, reliable distributed systems that process telemetry events.
- Scale and optimize metrics pipelines using OpenTelemetry, vmagent, and Grafana Mimir, including aggregation, sampling, and unused-metric filtering.
- Modernize logging and tracing pipelines using OpenTelemetry and improve storage capabilities with technologies such as OpenSearch.
- Develop secure, reliable query systems for engineers and AI agents, incorporating rate limiting, access controls, isolation, and caching.
- Build self-service tools for telemetry cost visibility and attribution, and controls for data granularity, sampling, and retention.
- Improve incident detection through anomaly detection, automated signal correlation, and enhanced monitoring.
- Align technical roadmaps with infrastructure and product teams, influence build-versus-buy decisions, and communicate architectural tradeoffs and investment priorities.
- Write maintainable, well-tested code, participate in design reviews, improve operational readiness, contribute to open-source projects, and mentor engineers.
View Full Description & ApplyYou'll be redirected to the employer's site
