Data Engineer

Posted 2024-11-07

View full description

💎 Seniority level: Middle, At least two years

📍 Location: Italy

🔍 Industry: AI solutions for highly regulated industries

🏢 Company: iGenius

🗣️ Languages: English

⏳ Experience: At least two years

🪄 Skills: PythonSQLAgileJenkinsAirflowNosqlCollaborationCI/CD

Requirements:

At least two years' proven experience in a data engineer role.
A degree in Computer Science, Applied Math, Informatics, Information Systems or similar.
Experience building processes supporting data transformation, data structure, metadata, dependency and workload management.
Advanced experience with relational databases, query authoring, SQL and NoSQL.
Experience building and optimizing data pipelines, architectures, and data sets.
Experience with Python or Scala, and distributed computing.
Hands-on experience with object-oriented design, coding, and testing patterns.
Knowledge of pipeline and workflow management tools like Airflow or Argo Workflows.
Experience with CI/CD tools such as Jenkins, Travis, Argo CD, Terraform.
Good understanding of Cloud and data models like data warehouse and data lake.
Experience with MLOps and HPC is a plus.

Responsibilities:

Create and maintain optimal data pipeline architectures.
Assemble large, complex data sets that meet functional and business requirements.
Identify, design, and implement improvements to internal practices, automating manual processes and optimizing data delivery.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources.

Apply

Related Jobs

Apply

🔥 Data Engineer, Blockchain

Posted 2024-11-15

📍 North America, South America, Europe

💸 100000 - 500000 USD per year

🔍 Web3, blockchain

🏢 Company: Edge & Node

A self-motivated, team member with keen attention to detail.
Proactive collaboration with team members and a willingness to adapt to a growing environment.
Familiarity and experience with Rust, particularly focusing on data transformation and ingestion.
A strong understanding of blockchain data structures and ingestion interfaces.
Experience in real-time data handling, including knowledge of reorg handling.
Familiarity with blockchain clients like Geth and Reth is a plus.
Adaptability to a dynamic and fully-remote work environment.
Rigorous approach to software development that reflects a commitment to excellence.

Develop and maintain data ingestion adapters for various blockchain networks and web3 protocols.
Implement data ingestion strategies for both historical and recent data.
Apply strategies for handling block reorgs.
Optimize the latency of block ingestion at the chain head.
Write interfaces with file storage protocols such as IPFS and Arweave.
Collaborate with upstream data sources, such as chain clients and tracing frameworks, and monitor the latest upstream developments.
Perform data quality checks, cross-checking data across multiple sources and investigating any discrepancies that arise.

Software DevelopmentBlockchainData StructuresRustCollaborationAttention to detail

Posted 2024-11-15

Apply

🔥 Lead Data Engineer

Posted 2024-11-07

📍 North America, Latin America, Europe

🔍 Data consulting

Bachelor’s degree in engineering, computer science or equivalent area.
5+ years in related technical roles such as data management, database development, and ETL.
Expertise in evaluating and integrating data ingestion technologies.
Experience in designing and developing data warehouses with various platforms.
Proficiency in building ETL/ELT ingestion pipelines with tools like DataStage or Informatica.
Cloud experience on AWS; Azure and GCP experience is a plus.
Proficiency in Python scripting; Scala is required.

Designing and developing Snowflake Data Cloud solutions.
Creating data ingestion pipelines and working on data architecture.
Ensuring data governance and security throughout customer projects.
Leading technical teams and collaborating with clients on data initiatives.

AWSLeadershipPythonSQLAgileETLOracleSnowflakeData engineeringSparkCollaboration

Posted 2024-11-07

Apply

🔥 Senior Data Engineer

Posted 2024-11-07

📍 Any European country

🧭 Full-Time

🔍 Software development

🏢 Company: Janea Systems

Proven experience as a data engineer, preferably with at least 3 or more years of relevant experience.
Experience designing cloud native solutions and implementations with Kubernetes.
Experience with Airflow or similar pipeline orchestration tools.
Strong Python programming skills.
Experience collaborating with Data Science and Engineering teams in production environments.
Solid understanding of SQL and relational data modeling schemas.
Preference for experience with Databricks or Spark.
Familiarity with modern data stack design and data lifecycle management.
Experience with distributed systems, microservices architecture, and cloud platforms like AWS, Azure, Google Cloud.
Excellent problem-solving skills and strong communication skills.

Develop and maintain data pipelines using Databricks, Airflow, or similar orchestration systems.
Design and implement cloud-native solutions using Kubernetes for high availability.
Gather product data requirements and implement solutions to ingest and process data for applications.
Collaborate with Data Science and Engineering teams to optimize production-ready applications.
Cultivate data from various sources for data scientists and maintain documentation.
Design modern data stack for data scientists and ML engineers.

AWSPythonSoftware DevelopmentSQLKubernetesAirflowAzureData scienceSparkCollaboration

Posted 2024-11-07

Apply

🔥 Lead/Senior Data Engineer

Posted 2024-11-07

📍 UK, EU

🔍 Consultancy

🏢 Company: The Dot Collective

Advanced knowledge of distributed computing with Spark.
Extensive experience with AWS data offerings such as S3, Glue, Lambda.
Ability to build CI/CD processes including Infrastructure as Code (e.g. terraform).
Expert Python and SQL skills.
Agile ways of working.

Leading a team of data engineers.
Designing and implementing cloud-native data platforms.
Owning and managing technical roadmap.
Engineering well-tested, scalable, and reliable data pipelines.

AWSPythonSQLAgileSCRUMSparkCollaborationAgile methodologies

Posted 2024-11-07

Apply

🔥 Data Engineer

Posted 2024-08-26

📍 Americas, EMEA, APAC

🔍 Crypto and blockchain technology

4+ years of work experience in relevant fields such as Data Engineer, DWH Engineer, or Software Engineer.
Experience with data warehouse technologies and relevant data modeling best practices like Presto, Athena, Glue, etc.
Experience building data pipelines/ETL and familiarity with design principles; knowledge of Apache Airflow is a plus.
Excellent SQL and data manipulation skills using frameworks like Spark/PySpark or similar.
Proficiency in a major programming language such as Scala, Python, or Golang.
Experience with business requirements gathering for data sourcing.

Build scalable and reliable data pipeline that collects, transforms, loads, and curates data from internal systems.
Augment data platform with data pipelines from select external systems.
Ensure high data quality for pipelines built and maintain auditability.
Drive data systems to approach real-time processing.
Support the design and deployment of a distributed data store as the central source of truth.
Build data connections to internal IT systems.
Develop and customize self-service tools for data consumers.
Evaluate new technologies and create prototypes for continuous improvements in data engineering.

PythonSQLETLAirflowData engineeringSpark

Posted 2024-08-26

Apply

🔥 Data Engineer, Data Services

Posted 2024-08-10

📍 Central EU or Americas

🧭 Full-Time

🔍 Real Estate Investment

🏢 Company: Roofstock👥 501-1000💰 $240.0m Series E on 2022-03-10🫂 on 2023-03-22Rental Property PropTech Marketplace Real Estate FinTech

BS or MS in a technical field: computer science, engineering or similar.
5+ years technical experience working with data.
5+ strong experience building scalable data services and applications using either SQL, Python, Java / Kotlin.
Deep understanding of microservices architecture and RESTful API development including gRPC, REST/SOAP, GraphQL.
Experience with AWS services including Messaging such as SQS, SNS, and familiarity with real-time data processing frameworks.
Significant experience building and deploying data-related infrastructure, robust data pipelines, and ETL/ELT code.
Strong understanding of data architecture and related problems.
Experience working on complex problems and distributed systems where scalability and performance are important.
Strong communication and interpersonal skills.
Independent work and effective collaboration with cross-functional teams.

Improve and maintain the data services platform.
Deliver high-quality data services promptly, ensuring data governance and integrity while meeting objectives and maintaining SLAs for data sharing across multiple products.
Develop effective architectures and produce key code components that contribute to the design, implementation, and maintenance of technical solutions.
Integrate a diverse network of third-party tools into a cohesive, scalable platform, optimizing code for enhanced scalability, performance, and readability.
Continuously improve system performance and reliability by diagnosing and resolving unexpected operational issues.
Ensure the team's work undergoes rigorous testing through repeatable, automated methods.
Support the data infrastructure and the rest of the data team in designing, implementing, and deploying scalable, fault-tolerant pipelines.

AWSDockerGraphQLPythonSQLAgileETLJavaKafkaKotlinSCRUMSnowflakeAirflowApache KafkaData engineeringgRPC

Posted 2024-08-10

Apply

🔥 Senior Data Engineer, Data Services

Posted 2024-08-10

📍 Central EU or Americas

🧭 Full-Time

🔍 Real estate investment

🏢 Company: Roofstock👥 501-1000💰 $240.0m Series E on 2022-03-10🫂 on 2023-03-22Rental Property PropTech Marketplace Real Estate FinTech

BS or MS in a technical field: computer science, engineering or similar.
8+ years technical experience working with data.
5+ years strong experience building scalable data services and applications using SQL, Python, Java/Kotlin.
Deep understanding of microservices architecture and RESTful API development.
Experience with AWS services including messaging and familiarity with real-time data processing frameworks.
Significant experience building and deploying data-related infrastructure and robust data pipelines.
Strong understanding of data architecture and related challenges.
Experience with complex problems and distributed systems focusing on scalability and performance.
Strong communication and interpersonal skills.
Independent worker able to collaborate with cross-functional teams.

Improve and maintain the data services platform.
Deliver high-quality data services promptly, ensuring data governance and integrity while meeting objectives and maintaining SLAs.
Develop effective architectures and produce key code components contributing to technical solutions.
Integrate a diverse network of third-party tools into a cohesive, scalable platform.
Continuously enhance system performance and reliability by diagnosing and resolving operational issues.
Ensure rigorous testing of the team's work through automated methods.
Support data infrastructure and collaborate with the data team on scalable data pipelines.
Work within an Agile/Scrum framework with cross-functional teams to deliver value.
Influence the enterprise data platform architecture and standards.

AWSDockerPythonSQLAgileETLSCRUMSnowflakeAirflowData engineeringgRPCRESTful APIsMicroservices

Posted 2024-08-10

Apply

🔥 Data Engineer

Posted 2024-07-18

📍 Africa, United Kingdom, Europe, Middle East

🧭 Full-Time

🔍 Sports and Digital Entertainment

4+ years of experience in a data engineering or similar role.
Excellent programming skills in Python and Spark (PySpark / Databricks).
2+ years' experience with Databricks and Azure data services.
Experience with other cloud-based data management environments (AWS, Google Cloud, etc.) is an advantage.
Experience working with Customer Data Platforms is a plus.
Knowledge of managing data quality, including monitoring and alerting.
Good understanding of application and database development lifecycles.
Experience with remote working and ideally with hyper-growth startups.

Building and managing a highly robust and scalable Data Lake/ETL infrastructure.
Creating a scalable data pipeline for streaming and batch processing.
Ensuring data integrity through fault-tolerant systems and automated data quality monitoring.
Continuously improving processes and optimizing performance and scalability.
Ensuring privacy and data security are prioritized.
Documenting the Data Platform stack comprehensively.
Partnering with business stakeholders and product engineering to deliver data products.
Collaborating with stakeholders to shape requirements and drive the data platform roadmap.

PythonAgileETLAzureData engineeringSparkDocumentation

Posted 2024-07-18

Apply

🔥 Data Engineer

Posted 2024-07-18

📍 Africa, United Kingdom, Europe, Middle East

🔍 Sports and Digital Entertainment

4+ years' of experience in a data engineering or similar role.
Excellent programming skills in Python and Spark (PySpark/Databricks).
2+ years’ experience working with Databricks and Azure data services.
Experience with other cloud data environments (AWS, Google Cloud, Hadoop, etc.) is a plus.
Experience with Customer Data Platforms is advantageous.
Active management of data quality including monitoring and alerting.
Familiarity with the application lifecycle from development to production.
Remote working experience is required; hyper-growth startup experience is a plus.

Design and manage a scalable Data Lake/ETL infrastructure and data pipeline.
Ensure fault-tolerant systems and processes with data integrity and automated monitoring.
Continuously improve data processes for performance and scalability.
Ensure data privacy and security standards are met.
Maintain up-to-date documentation for the Data Platform stack.
Collaborate with business stakeholders and engineering to deliver data products.

PythonAgileETLKafkaAzureData engineeringSpark

Posted 2024-07-18

Apply

🔥 Data Engineer

Posted 2024-07-18

📍 Africa, United Kingdom, Europe, Middle East

🧭 Full-Time

🔍 Sports and Digital Entertainment

4+ years' of experience in a data engineering or similar role.
Excellent programming language skills in Python and Spark (PySpark / Databricks).
2+ years’ experience working with Databricks and Azure data services.
Experience with cloud-based data management environments like AWS, Google Cloud, Hadoop, Snowflake, Spark, Storm, Kafka is an advantage.
Experience managing data quality including monitoring and alerting.
Good understanding of the application and database development lifecycle.
Remote working experience is a must; hyper-growth startup experience is a strong plus.

Design and manage a highly robust and scalable Data Lake/ETL infrastructure and scalable data pipelines for streaming and batch processing.
Ensure fault-tolerant systems and processes with a high priority on data integrity, supported by automated quality monitoring and alerting.
Continuously seek improvements through fixing recurring problems, delivering helpful features, and optimizing for performance and scalability.
Maintain top-class and up-to-date documentation for the entire Data Platform stack.
Partner with business stakeholders and product engineering to deliver high-value data products and understand requirements.

PythonETLAzureData engineeringSpark

Posted 2024-07-18

Apply

Data Engineer

Requirements:

Responsibilities:

Related Jobs

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities