Principal Data Engineer
New
O
OctusCredit Intelligence
Remote - USFull-TimePrincipal
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 8+ years
- Required Skills
- AWSDockerPythonSQLApache AirflowETLCI/CDTerraform
Requirements
- Strong foundation in software engineering principles, including SOLID design, modularity, and scalability.
- Expert proficiency in Python for data pipeline and automation development.
- Advanced SQL skills and experience optimizing complex queries and data models.
- Proven experience designing and maintaining cloud-native data pipelines on AWS (e.g., MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, Redshift, etc.).
- Experience with data warehousing or lakehouse technologies (Redshift, Snowflake, Databricks, etc.).
- Experience implementing and managing Terraform or similar IaC frameworks.
- Strong understanding of data ingestion, transformation, and orchestration tools and patterns, including those used in AI/ML pipelines.
- Familiarity with CI/CD pipelines, automated testing, and modern DevOps practices.
- 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions.
- Extensive experience in a technical leadership capacity, including mentoring and leading complex data infrastructure projects end-to-end.
- Familiarity with containerization (Docker) and workflow orchestration best practices.
- Excellent communication, collaboration, and problem-solving skills.
Responsibilities
- Lead the overarching technical strategy for the data platform, ensuring alignment between data infrastructure and long-term business goals.
- Lead the design and development of data ingestion and transformation pipelines, ensuring scalability, efficiency, and reliability across diverse data sources.
- Serve as a foundational technical leader and mentor for senior engineers within the data platform team, guiding architecture, design, and implementation decisions.
- Architect and manage data pipelines and orchestration workflows using AWS services such as MWAA (Airflow), Lambda, ECS, and SQS.
- Implement and maintain infrastructure as code (IaC) using Terraform, ensuring reproducibility and compliance with cloud standards.
- Partner with data analysts, scientists, and backend engineers to ensure data consistency, discoverability, and reliability.
- Apply best practices in data modeling, schema design, and ETL/ELT processes for high-volume structured and semi-structured data.
- Ensure data quality and lineage through automated testing, monitoring, and alerting.
- Promote continuous improvement through code reviews, observability practices, and team-wide knowledge sharing.
View Full Description & ApplyYou'll be redirected to the employer's site
