Principal Data Engineer

View full description

Requirements:

• Extensive experience of designing and building end to end data solutions (10 years +)
• Experience of carrying out data engineering design and build activities using agile working practices
• Experience of Databricks solutions, Databricks administration and pyspark
• Data Factory/Synapse Workspace – for building data pipelines or synapse analytics pipelines
• Data Lake – Delta Lake design pattern implementation experience in Azure Data Lake Gen2
• Synapse Warehouse/Analytics – Experience in Synapse data mappings, external tables, schema creation from SSMS, knowledge on how Synapse pool works behind the scenes
• Azure Active Directory – for Managed identities creation and usage or for generating service principles for authentication and authorization
• Version Control – Experience in building Data Ops i.e., CICD pipelines in Azure DevOps with managed identity
• Unit Testing – Experience in writing unit tests for data pipelines
• Data Architecture – Knowledge or experience in implementing, Kimball style Data Warehouse
• Data Quality – Experience in applying Data Quality rules within Azure Data Flow Activities
• Data Transformation – Extensive hands on with Azure Data Flow Activities for Cleansing, transforming, validation and quality checks
• Azure Cloud – Knowledge and confidence in effective communication on Azure Cloud Subscriptions

Responsibilities:

• Create or guide the low-level design of data solutions
• Be responsible for the quality of the overall data platform(s)
• Be responsible for coding standards, low level design and ingestion patterns for the data platform(s)
• Develop high complexity, secure, governed, high quality, efficient data pipelines
• Set the standards and ensure that data is cleansed, mapped, transformed and optimised for storage
• Design and build of data observability and data quality by design into all Data pipelines
• Build solutions that pipe transform data into data lake storage areas, physical database models and reporting structures across data lake, data warehouse, business intelligence systems and analytics applications
• Build physical data models that are appropriately designed to meet business needs and optimise storage requirements
• Carry out unit testing of own code, peer testing of others code
• Ensure that effective, and appropriate documentation that brings transparency and understandability are in place for all content on the data platform(s)
• Coach and mentor Senior Data Engineers, Data engineers & Associate Data Engineers
• Create high complexity BI solutions