Senior Data Engineer, Emerging Products
New
N
NinjaOneData platform
We are flexible on remote working from home, if you are located in the USA and reside in one of the following states - CA, CO, CT, FL, GA, IL, KS, MA, MD, ME, NJ, NC, NY, OH, OR, TN, TX, VA, and WA.Full-TimeSenior
Salary140,000 - 200,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 10+ years of experience in data engineering with a strong focus on data platform and distributed systems.
- Required Skills
- PythonApache AirflowApache KafkaSparkDatabricks
Requirements
- Have 10+ years of data engineering experience focused on data platforms and distributed systems.
- Have hands-on experience building and managing lakehouse architectures, including Medallion layering patterns.
- Have deep expertise in Apache Kafka, Apache Spark, and Apache Airflow for streaming and batch pipeline development.
- Be proficient in open table formats such as Iceberg, Delta Lake, or Hudi, and evaluate trade-offs among formats.
- Have experience with Databricks for large-scale data processing and analytics workloads.
- Be proficient in Python and SQL.
- Have a bachelor’s degree in Computer Science, Computer Engineering, Information Technology, or equivalent work experience; preferred.
- Hands-on experience with Trino or Starburst is a standout qualification.
- Familiarity with hybrid streaming/batch architectures and event-driven data systems is a standout qualification.
- Experience designing Gold-layer datasets for business intelligence and self-serve analytics is a standout qualification.
Responsibilities
- Design and implement Medallion-layer data architectures with Bronze, Silver, and Gold layers.
- Build structured ingestion pipelines and curated data layers for analytics and data science workloads.
- Build and maintain scalable streaming and batch pipelines using Kafka, Spark, and Airflow.
- Manage Iceberg, Delta, and Hudi table formats, selecting formats for workloads and maintaining performance and reliability.
- Optimize large-scale analytics queries in Trino/Starburst and Databricks environments.
- Monitor pipeline health and troubleshoot data quality and performance issues.
- Improve the reliability and efficiency of the data platform.
- Partner with data scientists, analysts, and product teams to understand data requirements and deliver platform solutions.
View Full Description & ApplyYou'll be redirected to the employer's site