Senior Data Engineer
Y
YassirTechnology Super App
Cairo, Egypt / Alexandria, Egypt / Cape Town, South Africa / Johannesburg, South Africa / Lagos, Nigeria / Nairobi, Kenya / Remote / Abu Dhabi, United Arab Emirates / IndiaFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonSQLApache AirflowGCPMongoDBNosqlScalaBigQueryPySpark
Requirements
- Proficiency in PySpark for batch and streaming data processing.
- Extensive experience with GCP services including Dataproc, Dataflow, DataStream, Dataplex, Pub/Sub, BigQuery, and Cloud Storage.
- Experience with NoSQL databases, preferably MongoDB.
- Strong programming skills in Scala or Python.
- Familiarity with Data Quality frameworks like Great Expectations.
- Proficiency with workflow management tools such as Airflow, Prefect, or Luigi.
- Solid understanding of Data Governance, Data Warehousing, and Data Modelling.
- Advanced SQL knowledge.
- Ability to effectively communicate and distill technical knowledge into clear, succinct, and visual messages.
Responsibilities
- Build a centralized data lake on GCP Data services by integrating diverse data sources throughout the enterprise.
- Develop, maintain, and optimize SPARK-powered batch and streaming data processing pipelines.
- Design and implement data validation and quality checks to ensure data accuracy, completeness, and consistency.
- Work with Data Science and Machine Learning teams on advanced analytics.
- Collaborate with cross-functional teams to extract insights and value from data.
- Collaborate with the product team to design, implement, and maintain data models for analytical use cases.
- Design, develop, and upkeep data dashboards using Looker Studio.
- Design and manage ETL/ELT processes ensuring data integrity, availability, and performance.
View Full Description & ApplyYou'll be redirected to the employer's site
