Senior Data Pipeline Engineer
New
J
JobgetherData engineering
Based in the United StatesFull-TimeSenior
Salary100,000 - 130,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years of experience in database engineering, data engineering, or ETL development
- Required Skills
- AWSApache AirflowDynamoDBAWS Lambda
Requirements
- Have 5+ years of experience in database engineering, data engineering, or ETL development, with ownership of production data pipelines.
- Demonstrate expert-level proficiency in Transact-SQL, including stored procedures, views, and database indexing.
- Have hands-on development experience with C#/.NET.
- Have experience designing and implementing ETL processes using SQL Server Integration Services (SSIS).
- Have experience with AWS cloud services and open table formats such as Apache Iceberg and Parquet.
- Have strong data quality management, troubleshooting, and root-cause analysis capabilities, particularly in production environments.
- Understand source control, CI/CD practices, and release and deployment workflows across development, QA, and production.
- Be able to mentor engineers, conduct technical reviews, and communicate complex technical concepts effectively.
- Be able to prioritize independently, exercise business judgment, and manage competing responsibilities.
- Be able to collaborate within a remote and distributed team.
- Be willing to take on additional responsibilities and special projects as required.
- Familiarity with Apache Airflow is preferred; AWS Lambda, DynamoDB, and Trino/Starburst are advantageous.
Responsibilities
- Architect, develop, and maintain ETL/ELT pipelines and C#/.NET pipeline services using SSIS, Apache Airflow, AWS Lambda, data enhancement services, and message-processing solutions.
- Support the cloud data platform, including Apache Iceberg ingestion and transformation pipelines, DynamoDB integrations, and query performance optimization using Trino/Starburst.
- Monitor end-to-end data quality, investigate production pipeline incidents, identify root causes, document findings, and implement corrective actions through successful data reprocessing.
- Lead onboarding of external and regulatory data sources, coordinating requirements and integrating them across ingestion and transformation layers.
- Contribute to data schema design, architecture decisions, and technical improvements across the data platform.
- Maintain standard operating procedures, technical documentation, and engineering best practices.
- Mentor Associate-level engineers, review their work, and support their technical development.
- Participate in release management and deployment across development, QA, and production environments.
- Partner with the Data Operations Manager on technical architecture, platform strategy, and roadmap decisions.
- Provide escalation-level support for business-critical pipeline incidents, including after-hours assistance when required.
View Full Description & ApplyYou'll be redirected to the employer's site
