Data Engineer (Spark)
New
A
AddeptoData engineering
Workplace type: remote; Country code: PLFull-TimeMiddle
Salary15,120 - 21,000 PLN per month
Apply NowOpens the employer's application page
Job Details
- Languages
- Fluent in English (at least C1 level).
- Experience
- At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance and data management processes.
- Required Skills
- AWSPythonApache AirflowJavaApache KafkaSparkScalaData modeling
Requirements
- Have at least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance, and data management processes.
- Have strong programming skills in Python or Java/Scala, including clean code and object-oriented design.
- Have hands-on experience with Spark, Cloudera, Airflow, Kafka, NiFi, Docker, and Iceberg.
- Understand dimensional data and data modeling techniques.
- Have experience implementing and deploying solutions in cloud environments.
- Have consulting experience, including direct client interaction and client management.
- Be able to work independently and take ownership of project deliverables.
- Be fluent in English at C1 level or above.
- Hold a bachelor’s degree in a technical or mathematical field.
- Experience with Kubeflow or MLflow is a plus.
- Familiarity with Databricks or dbt is a plus.
Responsibilities
- Develop and maintain a high-performance data processing platform for automotive data.
- Design and implement pipelines for large volumes of streaming and batch data.
- Optimize data ingestion, processing, and storage workflows using technologies such as Spark, Cloudera, and Airflow.
- Use Iceberg to manage structured and unstructured data.
- Collaborate with cross-functional teams to understand data requirements and integrate data sources.
- Monitor and troubleshoot the platform to support availability, performance, and data-processing accuracy.
- Use AWS for infrastructure management and scaling processing workloads.
- Write and maintain Python, Java, or Scala code for data processing and automation.
View Full Description & ApplyYou'll be redirected to the employer's site
