Senior Data Engineer

View full description

Requirements:

• 5+ years of professional experience using Python, Java, or Scala for data processing (Python preferred)
• Experience with writing data processing pipelines and with cloud platforms like AWS, GCP, or Azure
• Experience with data pipeline orchestration tools like Apache Airflow (preferred), Dagster or Prefect
• Deep understanding of data warehousing strategies
• Experience with transformation tools like dbt to manage data transformation in your data pipelines
• Some experience in managing infrastructure with IaC tools like Terraform
• Stay updated with industry trends, emerging technologies, and best practices in data engineering
• Improve, manage, and teach standards for code maintainability and performance in code submitted and reviewed
• Ship large features independently, generate architecture recommendations with the ability to implement them
• Strong communicator that can describe complex topics in a simple way to a variety of technical and non-technical stakeholders.

Responsibilities:

• Design and develop scalable, reliable data pipelines using the Google Cloud stack.
• Ingest, process, and store structured and unstructured data from various sources into our data-lakes and data warehouses.
• Optimise data pipelines for cost, performance and scalability.
• Implement and maintain data governance frameworks, ensuring data accuracy, consistency, and compliance.
• Monitor and troubleshoot data pipeline issues, implementing proactive measures for reliability and performance.
• Mentor and provide technical guidance to other engineers working with data.
• Partner with Product, Engineering & Data Science teams to operationalise new solutions.