Senior Data Engineer

Posted about 1 month agoViewed

View full description

💎 Seniority level: Senior, 3+ years related work experience

📍 Location: South Africa, Mauritius, Kenya, Nigeria, SAST, NOT STATED

🔍 Industry: Technology, Marketplaces

⏳ Experience: 3+ years related work experience

🪄 Skills: AWSPostgreSQLPythonSQLETLGitCI/CD

Requirements:

BSc degree in Computer Science, Information Systems, Engineering, or related technical field or equivalent work experience.
3+ years related work experience.
Minimum of 2 years experience building and optimizing ‘big data’ data pipelines, architectures and maintaining data sets.
Experienced in Python.
Experienced in SQL (PostgreSQL, MS SQL).
Experienced in using cloud services: AWS, Azure or GCP.
Proficiency in version control, CI/CD and GitHub.
Understanding/experience in Glue and PySpark highly desirable.
Experience in managing data life cycle.
Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements.
Ability to maintain and optimise processes supporting data transformation, data structures, metadata, dependency and workload management.
Good understanding of data management principles - data quality assurance and governance.
Strong analytical skills related to working with unstructured datasets.
Understanding of message queuing, stream processing, and highly scalable ‘big data’ datastores.
Strong attention to detail.
Good communication and interpersonal skills.

Responsibilities:

Suggest efficiencies and execute on implementation of internal process improvements in automating manual processes.
Implement enhancements and new features across data systems.
Improve streamline processes within data systems with support from Senior Data Engineer.
Test CI/CD process for optimal data pipelines.
Assemble large, complex data sets that meet functional / non-functional business requirements.
Highly efficient in ETL processes.
Develop and conduct unit tests on data pipelines as well as ensuring data consistency.
Develop and maintain automated monitoring solutions.
Support reporting and analytics infrastructure.
Maintain data quality and data governance as well as upkeep of overall maintenance of data infrastructure systems.
Maintain data warehouse and data lake metadata, data catalogue, and user documentation for internal business users.
Ensure best practice is implemented and maintained on database.

Apply

Related Jobs

Apply

🔥 Senior Data Engineer

Posted about 4 hours ago

📍 Germany, Austria, Italy, Spain, Portugal

🧭 Full-Time

🔍 Digital Solutions for Financial and Real Estate Industries

🏢 Company: PriceHubble👥 101-250💰 Non-equity Assistance about 3 years agoArtificial Intelligence (AI)PropTech Big Data Machine Learning Analytics Real Estate

🔧 Requirements

3+ years experience building and maintaining production data pipelines
Proficient in working with geospatial data (bonus)

💡 Responsibilities

Work with backend engineers and data scientists to turn raw data into trusted insights
Navigate cost-value trade-offs to deliver value to customers
Develop solutions that work in over 10 countries
Lead a project from concept to launch
Drive the team to deliver high-quality products, services, and processes
Improve the performance, data quality, and cost-efficiency of data pipelines
Maintain and monitor the data systems

PostgreSQLPythonSQLApache AirflowETLData engineering

Posted about 4 hours ago

Apply

🔥 Senior Data Engineer

Posted 2 days ago

📍 United States, Canada

🧭 Full-Time

🔍 B2B SaaS

🏢 Company: Sanity

🔧 Requirements

4+ years of experience building data pipelines at scale
Deep expertise in SQL, Python, and Node.js/TypeScript
Production experience with Airflow and RudderStack
Track record of building reliable data infrastructure

💡 Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines
Collaborate to implement and scale product telemetry
Establish best practices for data ingestion and transformation
Monitor and optimize data pipeline performance

Node.jsPythonSQLApache AirflowETLTypeScript

Posted 2 days ago

Apply

🔥 Senior Data Engineer

Posted 2 days ago

📍 United States, Canada

🧭 Full-Time

🔍 E-commerce

🔧 Requirements

Bachelor's or Master's degree in Computer Science or related field
5+ years of experience in data engineering
Strong proficiency in SQL and database technologies
Experience with data pipeline orchestration tools
Proficiency in programming languages like Python and Scala
Hands-on experience with AWS cloud data services
Familiarity with big data frameworks like Apache Spark
Knowledge of data modeling and warehousing
Experience implementing CI/CD for data pipelines
Real-time data processing architectures experience

💡 Responsibilities

Design, develop, and maintain ETL/ELT pipelines
Optimize data architecture and storage solutions
Work with AWS for scalable data solutions
Ensure data quality, integrity, and security
Collaborate with cross-functional teams
Monitor and troubleshoot data workflows
Create APIs for analytical information

AWSPostgreSQLPythonSQLApache AirflowETLKafkaMySQLSnowflakeCI/CDScala

Posted 2 days ago

Apply

🔥 Senior Data Engineer

Posted 3 days ago

📍 Europe, APAC, Americas

🧭 Full-Time

🔍 Software Development

🏢 Company: Docker👥 251-500💰 $105,000,000 Series C almost 3 years agoDeveloper Tools Developer Platform Information Technology Software

🔧 Requirements

4+ years of relevant industry experience
Experience with data modeling and building scalable pipelines
Proficiency with Snowflake or BigQuery
Experience with data governance and security controls
Experience creating ETL scripts using Python and SQL
Familiarity with a cloud ecosystem: AWS/Azure/Google Cloud
Experience with Tableau or Looker

💡 Responsibilities

Manage and develop ETL jobs, warehouse, and event collection tools
Build and manage the Central Data Model for reporting
Integrate emerging methodologies and technologies
Build data pipelines for ML and AI projects
Contribute to SOC2 compliance across the data platform
Document technical architecture

PythonSQLETLSnowflakeAirflowData engineeringData visualizationData modeling

Posted 3 days ago

Apply

🔥 Senior Data Engineer

Posted 6 days ago

📍 United States, Canada

🧭 Full-Time

🔍 Software Development

🏢 Company: BioRender👥 101-250💰 $15,319,133 Series A almost 2 years agoLife Science Graphic Design Software

🔧 Requirements

7+ years of data engineering experience of relevant industry experience
Expertise working with Data Warehousing platforms (AWS RedShift or Snowflake preferred) and data lake / lakehouse architectures
Experience with Data Streaming platforms (AWS Kinesis / Firehose preferred)
Expertise with SQL and programming languages commonly used in data platforms (Python, Spark, etc)
Experience with data pipeline orchestration (e.g., Airflow) and data pipeline integrations (e.g. Airbyte, Stitch)

💡 Responsibilities

Building and maintaining the right architecture and tooling to support our data science, analytics, product, and machine learning initiatives.
Solve complex architectural problems
Translate deeply technical designs into business appropriate representations as well as analyze business needs and requirements ensuring implementation of data services directly correlates to the strategy and growth of the business

AWSPythonSQLApache AirflowSnowflakeData engineeringSparkData modeling

Posted 6 days ago

Apply

🔥 Senior Data Engineer (m/w/d)

Posted 7 days ago

📍 Germany

🧭 Full-Time

🔍 Insurtech

🏢 Company: Getsafe

🔧 Requirements

4+ years of experience in creating data pipelines using SQL/Python/Airflow
Experience designing Data Mart and Data Warehouse
Experience with cloud infrastructure, including Terraform

💡 Responsibilities

Analyze, design, develop, and deliver Data Warehouse solutions
Create ETL/ELT pipelines using Python and Airflow
Design, develop, maintain and support Data Warehouse & BI platform

PythonSQLApache AirflowETLTerraform

Posted 7 days ago

Apply

🔥 Senior Data Engineer

Posted 11 days ago

📍 UK

🧭 Full-Time

🔍 Technology, Data Engineering

🏢 Company: Aker Systems👥 101-250💰 over 4 years agoCloud Data Services Business Intelligence Analytics Software

🔧 Requirements

Data pipeline development using processing technologies
Experience in Public Cloud services, especially AWS
Configuring and tuning Relational and NoSQL databases
Programming with Python

💡 Responsibilities

Code, test, and document data pipelines
Conduct database design
Expand data platform capabilities
Perform data analysis and root cause analysis

AWSPythonData modeling

Posted 11 days ago

Apply

🔥 Senior Data Engineer

Posted 11 days ago

📍 Bulgaria

🧭 Full-Time

🔍 Software Development

🏢 Company: Dreamix Ltd.

🔧 Requirements

5 years of relevant experience in data engineering
Bachelor's degree in Computer Science or related field
Strong proficiency in Python
Familiarity with big data technologies such as Hadoop, Spark, and Kafka
Experience with cloud platforms (AWS, Azure, Google Cloud)
Understanding of data warehousing concepts
Experience with databases like SQL Server, Oracle, or PostgreSQL

💡 Responsibilities

Design, develop, and maintain scalable data pipelines
Collaborate with data scientists and analysts
Utilize Python for data processing and analysis
Implement ETL processes
Troubleshoot data-related issues
Implement data security best practices
Develop and maintain API integrations

AWSPostgreSQLPythonETLHadoopKafkaOracleAzureSparkData modeling

Posted 11 days ago

Apply

🔥 Senior Data Engineer

Posted 11 days ago

📍 Poland

🧭 Full-Time

🔍 Data Engineering

🏢 Company: Softeta

🔧 Requirements

Advanced degree in computer science, mathematics, or related field
Proven work experience as a data engineer (5+ years)
Proficiency with AirFlow, DBT, or similar products
Strong knowledge of data structures and modeling
CI/CD pipeline and MLOPs experience advantageous
Experience with cloud data platforms, particularly GCP/BigQuery

💡 Responsibilities

Create and maintain pipeline architectures in AirFlow and DBT
Assemble large and/or complex datasets
Improve processes for scale, delivery, and automation
Maintain and improve data warehouse structure
Communicate technical details to stakeholders
Investigate and resolve anomalies in data

SQLGCPAirflowData engineeringCI/CDData modeling

Posted 11 days ago

Apply

🔥 Senior Data Engineer, Capacity Systems & Analytics

Posted 11 days ago

📍 United States

🧭 Full-Time

💸 175000.0 - 205000.0 USD per year

🔍 Software Development

🏢 Company: CoreWeave💰 $642,000,000 Secondary Market about 1 year agoCloud Computing Machine Learning Information Technology Cloud Infrastructure

🔧 Requirements

Hands-on experience applying Kimball Dimensional Data Modeling principles to large datasets.
Expertise in working with analytical table/file formats, including Iceberg, Parquet, Avro, and ORC.
Proven experience optimizing MPP databases (StarRocks, Snowflake, BigQuery, Redshift).
Minimum 5+ years of programming experience in Python or Scala.
Advanced SQL skills, with a strong ability to write, optimize, and debug complex queries.
Hands-on experience with Airflow for batch orchestration distributed computing frameworks like Spark or Flink.

💡 Responsibilities

Develop and maintain data models, including star and snowflake schemas, to support analytical needs across the organization.
Establish and enforce best practices for dimensional modeling in our Lakehouse.
Engineer and optimize data storage using analytical table/file formats (e.g., Iceberg, Parquet, Avro, ORC).
Partner with BI, analytics, and data science teams to design datasets that accurately reflect business metrics.
Tune and optimize data in MPP databases such as StarRocks, Snowflake, BigQuery, or Redshift.
Collaborate on data workflows using Airflow, building and managing pipelines that power our analytical infrastructure.
Ensure efficient processing of large datasets through distributed computing frameworks like Spark or Flink.

AWSDockerPythonSQLCloud ComputingETLKubernetesSnowflakeAirflowAlgorithmsApache KafkaData engineeringData StructuresREST APISparkCommunication SkillsAnalytical SkillsCollaborationCI/CDRESTful APIsDevOpsTerraformProblem-solving skillsJSONScalaData visualizationAnsibleData modelingData analyticsDebugging

Posted 11 days ago

Apply