Senior Data Engineer
L
Lumicode Sp. z o.o. (Pentacomp Group)IT / Insurance
Warszawa, PLContractSenior
Salary23520 - 30240 PLN per hour b2b currencySource=original; 5075 - 6525 CHF per hour b2b currencySource=conversion; 5376 - 6912 EUR per hour b2b currencySource=conversion; 4628 - 5950 GBP per hour b2b currencySource=conversion; 6124 - 7874 USD per hour b2b currencySource=conversion
Apply NowOpens the employer's application page
Job Details
- Languages
- Polish (C2), English (B2)
- Required Skills
- PythonCI/CDPySpark
Requirements
- Very good knowledge of Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
- Strong understanding of the Medallion (Bronze/Silver/Gold) architecture.
- Experience in data integration and processing from APIs, JSON, and files.
- Practical knowledge of Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Storage.
- Experience with Databricks Workflows/Jobs; Delta Live Tables/Lakeflow is a plus.
- Familiarity with schema enforcement, schema evolution, and Unity Catalog.
- Experience implementing data quality rules, validation, and reconciliation.
- Background in metadata management, data lineage, and API-based ingestion/export.
- Proven experience with performance optimization and data partitioning techniques.
- Experience in configuring CI/CD for notebooks and code, with automated data testing.
- Experience working in regulated environments requiring compliance and high data quality.
- Proficiency in English sufficient for professional communication.
Responsibilities
- Design, develop, and maintain data pipelines using Azure Databricks, Python, PySpark, Spark SQL, and Delta Lake.
- Implement solutions following the Bronze, Silver, and Gold (Medallion) architecture.
- Ingest and process data from APIs, JSON files, Azure Storage, and enterprise systems.
- Develop and maintain data quality, validation, reconciliation, monitoring, and automated testing mechanisms.
- Optimize data processing performance, partitioning, and scalability.
- Implement CI/CD, automated testing, error handling, and reprocessing processes.
- Ensure traceability, auditability, and reproducibility of processes in regulated environments.
- Collaborate with business and technical teams on data platform design and development.
View Full Description & ApplyYou'll be redirected to the employer's site
