Software Data Engineer (AWS)
New
J
JobgetherManufacturing data
Based in IrelandFull-Time
SalaryAt least 29,500 EUR per year
Apply NowOpens the employer's application page
Job Details
- Required Skills
- AWSPythonSQLSpark
Requirements
- Hold a master’s degree in Computer Science, Computer Engineering, or a related technical discipline; a PhD is a plus.
- Have strong programming and data engineering skills, particularly in Python and SQL.
- Have hands-on experience building streaming data pipelines using Kafka/MSK and Flink or Spark Streaming.
- Have practical experience with AWS EKS, Fargate, Lambda, S3, Aurora/RDS PostgreSQL, DynamoDB, and VPC.
- Understand big data technologies such as Spark, Hive, and HDFS, and lakehouse formats such as Iceberg and Parquet.
- Have experience with Git, CI/CD, Docker, Terraform or other Infrastructure as Code tools, REST APIs, and Agile/Scrum methodologies.
- Understand scalable data architectures, production data pipelines, data quality, monitoring, and operational best practices.
- Be able to collaborate with cross-functional and international teams and work independently with technical and business stakeholders.
Responsibilities
- Design and develop streaming data pipelines using Kafka/MSK, Flink or Spark Streaming, and routing to curated data storage.
- Build batch transformation pipelines in a medallion lakehouse architecture on Amazon S3 using formats such as Iceberg and Parquet.
- Design and manage storage across relational databases, NoSQL technologies, and object-based lakehouse storage.
- Design, deploy, and maintain AWS infrastructure for data and machine learning workloads.
- Refactor and optimize solutions and take them from prototype through production using software engineering, CI/CD, containerization, and deployment practices.
- Collaborate with international teams to gather requirements, contribute to solution design, and define data loading and ingestion processes.
- Document technical developments, perform integration testing, and manage deployments in AWS environments.
- Monitor production pipelines and maintain observability, data quality, performance, reliability, and maintainability.
- Investigate user-reported issues and improve data ingestion processes in an application maintenance and support environment.
View Full Description & ApplyYou'll be redirected to the employer's site
