Data Engineer
New
Listing locations: PolandFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonSQLAgileETLApache KafkaAzureCI/CDDatabricksPySpark
Requirements
- Programming: Python/PySpark, SQL
- Proficient in building robust data pipelines using Databricks Spark
- Experienced in dealing with large and complex datasets
- Knowledgeable about building data transformations modules organized as libraries (Python packages)
- Familiar with Databricks Delta optimization techniques (partitioning, z-ordering, compaction, etc.)
- Experienced in developing CI/CD pipelines
- Experienced in leveraging event brokers (Kafka /Event Hubs / Kinesis)
- Understanding of basic networking concepts
- Familiar with Agile Software Development methodologies (Scrum)
Responsibilities
- Develop reusable, metadata-driven data pipelines
- Automate and optimize any data platform related processes
- Build integrations with data sources and data consumers
- Add data transformation methods to shared ETL libraries
- Write unit tests
- Develop solutions for the Databricks data platform monitoring
- Proactively resolve any performance or quality issues in ETL processes
- Cooperate with infrastructure engineering team to set up cloud resources
- Contribute to data platform wiki / documentation
- Perform code reviews and ensures code quality
- Initiate and implements improvements to the data platform architecture
View Full Description & ApplyYou'll be redirected to the employer's site
