Senior Big Data Engineer
New
T
TQLO SP. Z O.O.Big Data
100% zdalnie; Country code: PLContractSenior
Salary160 - 190 PLN per hour
Apply NowOpens the employer's application page
Job Details
- Languages
- Pl C1, en C1
- Required Skills
- PythonSQLMicrosoft AzureSparkData modeling
Requirements
- Have senior-level experience designing and developing Big Data or Data Engineering solutions.
- Have very good knowledge of SQL and data modeling.
- Have advanced Apache Spark knowledge, especially the DataFrame API, Spark SQL, and performance optimization.
- Have practical experience using Python for Data Engineering.
- Have experience designing, building, and optimizing ETL/ELT pipelines.
- Understand processing structured and semi-structured large datasets.
- Have practical knowledge of Data Lake architecture, including Parquet, Delta Lake, and data partitioning.
- Have experience with cloud platforms, especially Microsoft Azure.
- Know Git and CI/CD processes.
- Have basic knowledge of statistics, data analysis, and data quality controls.
- Be able to diagnose technical problems and optimize solution performance.
- Be able to work with technical and business stakeholders.
- Be available immediately or within 14 days.
- Azure Databricks experience is a plus.
- Experience with Machine Learning projects is a plus.
Responsibilities
- Design, build, and develop scalable ETL/ELT data pipelines in Microsoft Azure.
- Optimize Apache Spark processing with the DataFrame API and Spark SQL for performance, stability, and reliability.
- Design and develop Data Lake solutions using Parquet, Delta Lake, and data partitioning.
- Implement data quality controls and optimize data processing.
- Maintain code using Git and automate deployments through CI/CD.
- Prepare and process data for analytics solutions and Machine Learning projects.
- Analyze and resolve technical issues involving performance and large-scale data processing.
- Consult with project stakeholders about technical and business requirements.
View Full Description & ApplyYou'll be redirected to the employer's site
