Senior Data Engineer (Spark)
New
A
AddeptoAI and Data Engineering
Warszawa, Kraków, Wrocław, Katowice, Gdańsk, PolandFull-TimeSenior
Salary4137 - 5627 GBP per month b2b currencySource=conversion; 5575 - 7582 USD per month b2b currencySource=conversion; 4558 - 6199 CHF per month b2b currencySource=conversion; 4834 - 6574 EUR per month b2b currencySource=conversion; 21000 - 28560 PLN per month b2b currencySource=original
Apply NowOpens the employer's application page
Job Details
- Languages
- English (at least C1 level)
- Experience
- At least 3 years
- Required Skills
- PythonSQLHadoopJavaAirflowSparkScala
Requirements
- At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance, and data management processes.
- Strong programming skills in Python (or Java/Scala), including clean code and OOP design.
- Hands-on experience with Big Data technologies like Spark, Cloudera, Data Platform, Airflow, Kafka, NiFi, Docker, and Iceberg.
- Excellent understanding of dimensional data and data modeling techniques.
- Experience implementing and deploying solutions in cloud environments.
- Consulting experience with excellent communication and client management skills, including prior experience directly interacting with clients.
- Ability to work independently and take ownership of project deliverables.
- Fluent in English (at least C1 level).
- Bachelor’s degree in technical or mathematical studies.
Responsibilities
- Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.
- Design and implement data pipelines that process large volumes of data in both streaming and batch modes.
- Optimize data workflows to ensure efficient data ingestion, processing, and storage using technologies such as Spark, Cloudera, and Airflow.
- Work with data lake technologies (e.g., Iceberg) to manage structured and unstructured data efficiently.
- Collaborate with cross-functional teams to understand data requirements and ensure seamless integration of data sources.
- Monitor and troubleshoot the platform, ensuring high availability, performance, and accuracy of data processing.
- Leverage cloud services (AWS) for infrastructure management and scaling of processing workloads.
- Write and maintain high-quality Python (or Java/Scala) code for data processing tasks and automation.
View Full Description & ApplyYou'll be redirected to the employer's site
