Senior Data Engineer
New
R
Railroad19, IncSoftware Development
United StatesFull-TimeSenior
Salary119,998 - 180,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years
- Required Skills
- PythonKafkaSparkDatabricks
Requirements
- 6+ years of proven enterprise-level experience in Python & Spark.
- Advanced experience in GCP BigQuery.
- Strong working knowledge of Apache Iceberg, Delta Lake, and Iceberg UniForm.
- Experience with Delta Sharing and Kafka/CDC pipelines.
- Specific work experience in Snowflake Horizon Catalog and Databricks Unity Catalog.
- Solid experience with data lake architecture and ingestion pipeline design.
- Ability to work cohesively with multiple cross-functional teams.
Responsibilities
- Design and implement the UniForm write layer using Delta and Iceberg dual metadata.
- Build GCS to BigQuery ingestion pipelines for structured operational datasets.
- Develop and implement robust data pipelines using Python and Spark.
- Implement Kafka-based CDC patterns for real-time and near-real-time ingestion.
- Develop data lineage, dependency tracking, and modular adapter code.
- Configure Snowflake Horizon external tables for zero-copy reads.
- Implement and certify Delta Sharing endpoints for Databricks consumers.
- Build governed access layer components including RBAC and connector registry entries.
View Full Description & ApplyYou'll be redirected to the employer's site
