Senior Data Engineer

Posted 16 days agoViewed

💎 Seniority level: Senior, 5+ years

📍 Location: South Africa

🔍 Industry: Retail

⏳ Experience: 5+ years

🪄 Skills: AWSPythonSQLApache AirflowETLGCPSnowflakeApache KafkaData engineeringSparkRESTful APIsData modeling

5+ years (preferred 8+ years) of proven experience as a Data Engineer or in a similar technical role, with a strong track record of building scalable solutions.
Expertise in programming languages such as Python, Java, or Scala.
Advanced proficiency with SQL and database optimization techniques.
Experience with data warehousing solutions (e.g. Databricks, Snowflake, Redshift, BigQuery, Synapse).
Experience with distributed systems and big data technologies like Hadoop, Spark, Flink, or similar.
Strong knowledge of cloud-based data platforms and their services (e.g. AWS, Azure, GCP).
Experience with ETL tools and frameworks (e.g. Apache Kafka, Apache Airflow, Apache Spark etc.).
Experience in designing and implementing data architectures that support large-scale data processing.
Ability to work effectively with cross-functional teams, including data scientists, analysts, and business stakeholders.
Experience leading and mentoring junior data engineers and other team members
Experience contributing to and shaping internal data science standards and best practices.
Strong problem-solving and critical thinking skills, with the ability to identify and resolve issues in data pipelines or infrastructure.
Strong interpersonal skills, with the ability to communicate effectively across technical and non-technical teams.
An innovative, collaborative mindset with excellent communication skills to work across teams.

Lead the design and implementation of scalable and secure data architectures to support analytics and machine learning initiatives.
Develop and optimise complex, high-volume data pipelines and databases to process structured and unstructured data in real time or batch.
Ensure data pipelines and databases are optimised for performance, security, availability, and scalability.
Work closely with data scientists and data analysts to ensure they have access to the right data and that data is formatted for their analysis needs.
Develop and enforce standards and best practices for data quality, documentation, quality control, security, and compliance across all data processes.
Evaluate and implement new tools, technologies, and frameworks to improve data systems and processes.
Serve as a technical leader, coach and mentor for junior team members and adjacent data and engineering teams.
Lead end-to-end data engineering projects that support strategic goals. This includes requirements gathering, technical deliverable planning, output quality control and stakeholder management.
Contribute to the development of the IOB data strategy.

Posted 3 months ago

📍 South Africa, Mauritius, Kenya, Nigeria

🔍 Technology, Marketplaces

🔧 Requirements

BSc degree in Computer Science, Information Systems, Engineering, or related technical field or equivalent work experience.
3+ years related work experience.
Minimum of 2 years experience building and optimizing ‘big data’ data pipelines, architectures and maintaining data sets.
Experienced in Python.
Experienced in SQL (PostgreSQL, MS SQL).
Experienced in using cloud services: AWS, Azure or GCP.
Proficiency in version control, CI/CD and GitHub.
Understanding/experience in Glue and PySpark highly desirable.
Experience in managing data life cycle.
Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements.
Ability to maintain and optimise processes supporting data transformation, data structures, metadata, dependency and workload management.
Good understanding of data management principles - data quality assurance and governance.
Strong analytical skills related to working with unstructured datasets.
Understanding of message queuing, stream processing, and highly scalable ‘big data’ datastores.
Strong attention to detail.
Good communication and interpersonal skills.

💡 Responsibilities

Suggest efficiencies and execute on implementation of internal process improvements in automating manual processes.
Implement enhancements and new features across data systems.
Improve streamline processes within data systems with support from Senior Data Engineer.
Test CI/CD process for optimal data pipelines.
Assemble large, complex data sets that meet functional / non-functional business requirements.
Highly efficient in ETL processes.
Develop and conduct unit tests on data pipelines as well as ensuring data consistency.
Develop and maintain automated monitoring solutions.
Support reporting and analytics infrastructure.
Maintain data quality and data governance as well as upkeep of overall maintenance of data infrastructure systems.
Maintain data warehouse and data lake metadata, data catalogue, and user documentation for internal business users.
Ensure best practice is implemented and maintained on database.

AWSPostgreSQLPythonSQLETLGitCI/CD

Posted 3 months ago