Senior Data Engineer (Databricks)
New
A
AddeptoData Engineering, AI
Warszawa, Krucza 50, Kraków, Aleksandra Lubomirskiego 20, Wrocław, Świdnicka 40, Katowice, -, Gdańsk, -, Warszawa, Country code: PLFull-TimeSenior
Salary21,000 - 28,560 PLN per month
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 5 years of commercial experience
- Required Skills
- PythonSQLMicrosoft Power BIAirflowApache KafkaAzureSparkdbtDatabricks
Requirements
- At least 5 years of commercial experience implementing, developing, or maintaining Big Data systems.
- Strong programming skills in Python: writing a clean code, OOP design.
- Strong SQL skills, including performance tuning, query optimization, and experience with data warehousing solutions.
- Experience in designing and implementing data governance and data management processes.
- Deep expertise in Big Data technologies, including Apache Airflow, Dagster, Databricks, and other modern data orchestration and transformation tools.
- Experience implementing and deploying solutions in cloud environments (with a preference for Azure).
- Knowledge of how to build and deploy Power BI reports and dashboards for data visualization.
- Excellent understanding of dimensional data and data modeling techniques.
- Consulting experience and the ability to guide clients through architectural decisions, technology selection, and best practices.
- Ability to work independently and take ownership of project deliverables.
- Familiarity with Spark, Azure Event Hub, or Kafka.
- Master’s or Ph.D. in Computer Science, Big Data, Mathematics, Physics, or a related field.
Responsibilities
- Design and optimize scalable data processing pipelines for both streaming and batch workloads using Big Data technologies such as Databricks, Apache Airflow, and Dagster.
- Architect and implement end-to-end data platforms, ensuring high availability, performance, and reliability.
- Lead the development of CI/CD and MLOps processes to automate deployments, monitoring, and model lifecycle management.
- Develop and maintain applications for aggregating, processing, and analyzing data from diverse sources, ensuring efficiency and scalability.
- Collaborate with Data Science teams on Machine Learning projects, including text/image analysis, feature engineering, and predictive model deployment.
- Design and manage complex data transformations using Databricks, DBT, and Apache Airflow, ensuring data integrity and consistency.
- Translate business requirements into scalable and efficient technical solutions while ensuring optimal performance and data quality.
- Ensure data security, compliance, and governance best practices are followed across all data pipelines.
View Full Description & ApplyYou'll be redirected to the employer's site
