Principal Software Engineer - Observability & Telemetry Data
New
S
SmartsheetSoftware Development
Can be located in our Bellevue, WA office, or you may work remotely from anywhere in the US where Smartsheet is a registered employerFull-TimePrincipal
Salary$222,500 — $257,500 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- 10+ years
- Required Skills
- AWSPythonSQLJavaKubernetesMLFlowGoScalaDatabricks
Requirements
- 10+ years of experience building and operating large-scale distributed systems, data platforms, or observability infrastructure.
- Experience at a Principal or Staff engineering level.
- Deep expertise in metrics, logs, and distributed tracing with experience in major backends like Datadog.
- Strong background in telemetry data engineering including Delta Lake, Apache Iceberg, Spark, or similar distributed processing.
- Practical experience with Databricks platform (jobs, clusters, Unity Catalog, Delta) and MLflow.
- Practical experience with OpenTelemetry collectors, semantic conventions, and tail-based sampling.
- Proficiency in pipeline engineering using FluentBit/Fluentd, Kinesis/Kafka, and search backend index design.
- Advanced AWS and Kubernetes expertise (EKS, ECS Fargate, EC2, Lambda, CloudWatch).
- 10+ years of programming experience in Go, Java, Python, or Scala with strong SQL skills.
- Experience with Infrastructure as Code (Terraform) and GitOps workflows (Flux or ArgoCD).
- Degree in Computer Science, Engineering, or equivalent practical experience.
Responsibilities
- Architect the end-to-end design for landing, modeling, and querying telemetry data.
- Define semantic conventions and schema standards to ensure correlation of data across service teams.
- Lead migration to OpenTelemetry instrumentation including collector architecture and sampling strategy.
- Integrate observability platforms with Databricks and MLflow for AI and agentic telemetry.
- Build analytics layers for reliability, incident metrics, and telemetry cost modeling.
- Architect high-volume collection and routing pipelines using FluentBit, Kinesis, and OTel collectors.
- Set cost architecture for telemetry data including ingest governance and storage tiering.
- Carry architectural decisions across organizational boundaries and mentor senior/mid-level engineers.
View Full Description & ApplyYou'll be redirected to the employer's site
