Data Engineer (Spark)
New
A
AddeptoAI Consulting
Warszawa, Krucza 50, Kraków, Centrum, Wrocław, Centrum, Katowice, -, Gdańsk, -, WarszawaFull-TimeMiddle
Salary2979 - 4137 GBP per month b2b currencySource=conversion; 3480 - 4834 EUR per month b2b currencySource=conversion; 3282 - 4558 CHF per month b2b currencySource=conversion; 4014 - 5575 USD per month b2b currencySource=conversion; 15120 - 21000 PLN per month b2b currencySource=original
Apply NowOpens the employer's application page
Job Details
- Languages
- English C1
- Experience
- 3 years
- Required Skills
- PythonSQLHadoopJavaAirflowSparkScala
Requirements
- At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance, and data management processes.
- Strong programming skills in Python (or Java/Scala), including writing clean code and OOP design.
- Hands-on experience with Big Data technologies like Spark, Cloudera, Data Platform, Airflow, Kafka, NiFi, Docker, and Iceberg.
- Excellent understanding of dimensional data and data modeling techniques.
- Experience implementing and deploying solutions in cloud environments.
- Consulting experience with excellent communication and client management skills.
- Ability to work independently and take ownership of project deliverables.
- Fluent in English (at least C1 level).
- Bachelor’s degree in technical or mathematical studies.
Responsibilities
- Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.
- Design and implement data pipelines that process large volumes of data in both streaming and batch modes.
- Optimize data workflows to ensure efficient data ingestion, processing, and storage using technologies such as Spark, Cloudera, and Airflow.
- Work with data lake technologies (e.g., Iceberg) to manage structured and unstructured data efficiently.
- Collaborate with cross-functional teams to understand data requirements and ensure seamless integration of data sources.
- Monitor and troubleshoot the platform, ensuring high availability, performance, and accuracy of data processing.
- Leverage cloud services (AWS) for infrastructure management and scaling of processing workloads.
- Write and maintain high-quality Python (or Java/Scala) code for data processing tasks and automation.
View Full Description & ApplyYou'll be redirected to the employer's site
