Principal Software Engineer - Observability & Telemetry Data
New
S
SmartsheetSoftware Engineering
Can be located in our Bellevue, WA office, or you may work remotely from anywhere in the US where Smartsheet is a registered employer.Full-TimePrincipal
Salary$222,500 — $257,500 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- 10+ years
- Required Skills
- AWSPythonSQLKubernetesGoTerraformDatabricksDistributed Systems
Requirements
- 10+ years of experience building and operating large-scale distributed systems, data platforms, or observability infrastructure.
- Proven experience at a Principal or Staff engineering level.
- Hands-on production ownership of metrics, logs, and distributed tracing.
- Demonstrated depth in large-scale data architecture, including open table formats (Delta Lake, Apache Iceberg) and distributed processing.
- Practical experience with Databricks (Unity Catalog, Delta) and MLflow.
- Practical experience with OpenTelemetry collectors, semantic conventions, and sampling strategies.
- Experience with high-volume telemetry pipelines (FluentBit/Fluentd) and streaming transport (Kinesis/Kafka).
- Advanced expertise in AWS services (EKS, ECS Fargate, EC2, Lambda, CloudWatch) and Kubernetes.
- 10+ years of programming experience with Go, Java, Python, or Scala, and strong SQL.
- Proficiency with Infrastructure as Code tools like Terraform and GitOps workflows (Flux or ArgoCD).
- Proven track record of setting technical direction and producing architectural documentation (RFCs, standards).
Responsibilities
- Architect the Telemetry Data Platform, owning end-to-end design for data landing, modeling, and querying.
- Define telemetry data models, semantic conventions, and schema standards across all service teams.
- Lead migration to OpenTelemetry-based instrumentation and define collector architecture.
- Integrate AI and agentic telemetry with Databricks and MLflow.
- Build analytics layers on telemetry to drive reliability, cost, and adoption reporting.
- Engineer high-volume collection and routing pipelines using FluentBit, Kinesis, and OTel.
- Own telemetry economics including ingest governance, cardinality control, and storage tiering.
- Carry architecture across organizational boundaries, partnering with Data and AI platform teams.
- Raise the technical bar through design/code reviews, architecture standards, and mentorship.
- Participate in production support and on-call rotations for complex issue resolution.
View Full Description & ApplyYou'll be redirected to the employer's site
