Data Engineer (Azure)
M
MOTIFEBackground screening
Kraków, Country code: PLContractSenior
Salary4759 - 5491 CHF per hour b2b currencySource=conversion; 5075 - 5856 EUR per hour b2b currencySource=conversion; 4334 - 5001 GBP per hour b2b currencySource=conversion; 5855 - 6756 USD per hour b2b currencySource=conversion; 21840 - 25200 PLN per hour b2b currencySource=original
Apply NowOpens the employer's application page
Job Details
- Languages
- En B2
- Required Skills
- PythonSQLKafkaSparkCI/CDPySpark
Requirements
- Strong experience with Python/PySpark and SQL.
- Hands-on experience building robust data pipelines using Databricks Spark.
- Experience processing large-scale datasets in production environments.
- Strong knowledge of Kafka or other streaming platforms such as Azure Event Hubs or Amazon Kinesis.
- Experience building streaming data integrations and event-driven data pipelines.
- Good understanding of Spark Structured Streaming.
- Familiarity with CDC (Change Data Capture) concepts; experience with Debezium is a plus.
- Strong knowledge of Databricks Delta optimization (partitioning, Z-ordering, compaction, etc.).
- Experience developing reusable Python libraries and packages.
- Hands-on experience with CI/CD pipelines.
- Good understanding of networking fundamentals.
- Familiarity with Agile/Scrum methodologies.
Responsibilities
- Build reusable, metadata-driven data pipelines.
- Automate and optimize data platform processes.
- Develop integrations with data sources and consumers.
- Extend shared ETL libraries with transformation methods.
- Write unit tests.
- Create monitoring solutions for the Databricks platform.
- Proactively address ETL performance and quality issues.
- Collaborate with infrastructure teams on cloud resources.
- Update data platform wiki and documentation.
- Conduct code reviews to ensure quality.
- Initiate and implement architecture improvements.
View Full Description & ApplyYou'll be redirected to the employer's site
