Lead Data Engineer (PySpark)

New

Within PolandFull-TimeLead

Salary not disclosed

Apply NowOpens the employer's application page

Job Details

Experience: 9+ years
Required Skills: PythonAgileArtificial IntelligenceMicrosoft Power BISCRUMCI/CDDevOpsAzure DevOpsPySpark

9+ years of experience in data engineering with AI exposure
Expertise in Azure Fabric and end-to-end Fabric experience
Knowledge of OneLake (Delta / OpenLake)
Advanced skills in Python, PySpark and SparkSQL
Proficiency in Cosmos DB (NoSQL API) and other Cosmos DB variants
Understanding of DF Gen2 and M-code
Capability to implement CI/CD pipelines using Azure DevOps or equivalent tools
Experience with generic Azure services and Power BI integration, semantic models and performance considerations
Background in Agile or Scrum environments
Strong ownership mindset and ability to lead by example
Excellent communication skills for technical and non-technical audiences
Good understanding of the financial domain

Lead the design, development and optimization of scalable data engineering solutions using Azure Fabric and cloud-native technologies
Own end-to-end data pipelines including ingestion, transformation, storage and analytics
Architect and implement solutions leveraging OneLake (Delta / OpenLake) and Fabric experiences
Develop and optimize PySpark, SparkSQL and Python-based data processing pipelines
Work with Cosmos DB (NoSQL API) and other Cosmos DB variants to support high-performance data access patterns
Implement and maintain CI/CD pipelines and promote DevOps best practices
Collaborate with data scientists, AI engineers and product stakeholders to enable AI-driven analytics and insights
Mentor and guide junior engineers, setting coding standards and best practices
Ensure data quality, security, governance and performance across platforms
Contribute to technical decision-making and solution architecture discussions

View Full Description & ApplyYou'll be redirected to the employer's site