Senior Data Engineer

Posted 2024-11-07

View full description

💎 Seniority level: Senior, at least 3 years

📍 Location: Any European country

🔍 Industry: Software development

🏢 Company: Janea Systems

🗣️ Languages: English

⏳ Experience: At least 3 years

🪄 Skills: AWSPythonSoftware DevelopmentSQLKubernetesAirflowAzureData scienceSparkCollaboration

Requirements:

Proven experience as a data engineer, preferably with at least 3 or more years of relevant experience.
Experience designing cloud native solutions and implementations with Kubernetes.
Experience with Airflow or similar pipeline orchestration tools.
Strong Python programming skills.
Experience collaborating with Data Science and Engineering teams in production environments.
Solid understanding of SQL and relational data modeling schemas.
Preference for experience with Databricks or Spark.
Familiarity with modern data stack design and data lifecycle management.
Experience with distributed systems, microservices architecture, and cloud platforms like AWS, Azure, Google Cloud.
Excellent problem-solving skills and strong communication skills.

Responsibilities:

Develop and maintain data pipelines using Databricks, Airflow, or similar orchestration systems.
Design and implement cloud-native solutions using Kubernetes for high availability.
Gather product data requirements and implement solutions to ingest and process data for applications.
Collaborate with Data Science and Engineering teams to optimize production-ready applications.
Cultivate data from various sources for data scientists and maintain documentation.
Design modern data stack for data scientists and ML engineers.

Apply

Related Jobs

Apply

🔥 Senior Data Engineer

Posted 2024-11-21

📍 Poland

🧭 Full-Time

🔍 Software development

🏢 Company: Sunscrapers sp. z o.o.

At least 5 years of professional experience as a data engineer.
Undergraduate or graduate degree in Computer Science, Engineering, Mathematics, or similar.
Excellent command in spoken and written English, at least C1.
Strong professional experience with Python and SQL.
Hands-on experience with DBT and Snowflake.
Experience in building data pipelines with Airflow or alternative solutions.
Strong understanding of various data modeling techniques like Kimball Star Schema.
Great analytical skills and attention to detail.
Creative problem-solving skills.
Great customer service and troubleshooting skills.

Modeling datasets and schemes for consistency and easy access.
Design and implement data transformations and data marts.
Integrating third-party systems and external data sources into data warehouse.
Building data flows for fetching, aggregation and data modeling using batch pipelines.

PythonSQLSnowflakeAirflowAnalytical SkillsCustomer serviceDevOpsAttention to detail

Posted 2024-11-21

Apply

🔥 Senior Data Engineer

Posted 2024-11-21

📍 Belgium, Spain

🔍 Hospitality industry

🏢 Company: Lighthouse

5+ years of professional experience using Python, Java, or Scala for data processing (Python preferred)
Experience with writing data processing pipelines and with cloud platforms like AWS, GCP, or Azure
Experience with data pipeline orchestration tools like Apache Airflow (preferred), Dagster or Prefect
Deep understanding of data warehousing strategies
Experience with transformation tools like dbt to manage data transformation in your data pipelines
Some experience in managing infrastructure with IaC tools like Terraform
Stay updated with industry trends, emerging technologies, and best practices in data engineering
Improve, manage, and teach standards for code maintainability and performance in code submitted and reviewed
Ship large features independently, generate architecture recommendations with the ability to implement them
Strong communicator that can describe complex topics in a simple way to a variety of technical and non-technical stakeholders.

Design and develop scalable, reliable data pipelines using the Google Cloud stack.
Ingest, process, and store structured and unstructured data from various sources into our data-lakes and data warehouses.
Optimise data pipelines for cost, performance and scalability.
Implement and maintain data governance frameworks, ensuring data accuracy, consistency, and compliance.
Monitor and troubleshoot data pipeline issues, implementing proactive measures for reliability and performance.
Mentor and provide technical guidance to other engineers working with data.
Partner with Product, Engineering & Data Science teams to operationalise new solutions.

PythonApache AirflowGCPJavaKafkaKubernetesAirflowData engineeringGrafanaPrometheusSparkCI/CDTerraformDocumentationCompliance

Posted 2024-11-21

Apply

🔥 Senior Data Engineer

Posted 2024-11-13

📍 United Kingdom

🔍 Payment and Financial Services

🏢 Company: Vitesse PSP

Experience with data pipeline orchestration tools such as Airflow, Luigi, or similar.
Experience with version control systems and CI/CD best practices using GitHub Actions.
Knowledge of data governance, privacy regulations (e.g., GDPR), and security best practices.
Proficiency with SQL and experience with distributed data processing tools such as Apache Spark.
Strong understanding of relational and NoSQL databases (e.g., PostgreSQL, MongoDB, Impala, Cassandra).
Experience with cloud infrastructure (Docker and Kubernetes, Terraform).
Experience in AWS platform architecture and cloud services.
A collaborative team member with Agile experience.
Familiarity with stream processing technologies (Kafka or Kinesis).
Nice to have: Experience with machine learning frameworks and pipelines, Delta Live Tables, Great Expectations, search optimizers (ElasticSearch/Lucene), REST alternatives (GraphQL, AsyncAPI), data science kits (Jupyter, Anaconda).

Design, build, and maintain scalable data pipelines and architectures to handle large volumes of structured and unstructured data.
Develop, enhance, and optimize ELT processes for ingesting, processing, and distributing data across multiple platforms in real time.
Build and manage data warehouses to support advanced analytics, reporting, and machine learning.
Implement data governance, quality checks, and validation processes to ensure the accuracy, consistency, observability, and security of data.
Optimize query performance and data storage costs through techniques like partitioning, indexing, vacuuming, and compression.
Build monitoring and alerting systems for data pipelines to proactively detect and resolve issues.
Optimize existing data pipelines for better performance, cost-efficiency, and scalability.
Work with data scientists, analysts, and business stakeholders to understand data needs.
Continuously research and integrate cutting-edge data technologies, tools, and practices to improve data engineering processes.
Team up with product engineers to identify, root cause, and resolve bugs.
Update documentation to help users navigate data products.
Ensure the data platform performs well and is always available for blue-chip clients.

AWSDockerGraphQLPostgreSQLSQLAgileElasticSearchKafkaKubernetesMongoDBTableauAirflowCassandraData engineeringElasticsearchNosqlSparkCI/CDTerraformDocumentation

Posted 2024-11-13

Apply

🔥 Senior Data Engineer

Posted 2024-11-07

📍 Canada, UK, US

🔍 Smart home technology

🏢 Company: ecobee

Proficiency in building data pipelines using Python and SQL.
Experience with Apache Spark, Apache Kafka, and Apache Airflow.
Experience with cloud-based data platforms, preferably GCP.
Familiarity with SQL-based operational databases.
Good understanding of machine learning lifecycle.
Strong experience in data modeling and schema design.
Experience with both batch and real-time data processing.
Excellent communication skills for collaborative work.

Design, build, and maintain scalable and efficient ETL/ELT pipelines.
Implement data extraction and processing solutions for analytics and machine learning.
Integrate diverse data sources into centralized data repositories.
Develop and maintain data warehousing solutions.
Monitor and optimize data workflows for performance and reliability.
Implement monitoring and logging for data pipelines.
Collaborate with cross-functional teams to understand data requirements.
Translate business requirements into technical specifications.
Implement data quality checks and cleansing procedures.
Create and maintain documentation for data pipelines.
Share knowledge and best practices within the team.
Architect data pipelines for massive IoT data streams.

LeadershipPythonSQLApache AirflowETLGCPIoTKafkaMachine LearningAirflowApache KafkaData engineeringSparkCommunication SkillsCollaboration

Posted 2024-11-07

Apply

🔥 Senior Data Engineer (Remote) - UK

Posted 2024-11-07

📍 UK

🧭 Full-Time

🔍 Knowledge management

🏢 Company: AlphaSights

5+ years of hands-on data engineering development.
Expert in Python and SQL.
Experience with SQL/NoSQL databases.
Experienced with AWS data services.
Proficiency in DataOps methodologies and tools.
Experience with CI/CD pipelines and managing containerized applications.
Proficiency in workflow orchestration tools such as Apache Airflow.
Experience in designing, building, and maintaining Data Warehouses.
Collaborative experience with cross-functional teams.
Knowledge of ETL frameworks and best practices.

Design, develop, deploy and support data infrastructure, pipelines and architectures.
Take ownership of reporting APIs, ensuring accuracy and timeliness for stakeholders.
Monitor dataflows and underlying systems, promoting necessary changes for scalability and performance.
Collaborate directly with stakeholders to translate business problems into data-driven solutions.
Mentor engineers within the technical guild and support team growth.

AWSPythonSQLApache AirflowETLAirflowData engineeringNosqlCI/CD

Posted 2024-11-07

Apply

🔥 Senior Data Engineer - Data Corpus

Posted 2024-11-07

📍 US, Germany, UK

🧭 Full-Time

🔍 Music

🏢 Company: SoundCloud

Senior Level Data Professional with a minimum of 4 years of experience (ideal 6+ years).
Experience with Cloud technologies, specifically GCP (required), with AWS/Azure as a plus.
Experience working with BigQuery and advanced SQL knowledge.
Proficiency in Python and Airflow.
Experience with big data at terabyte/petabyte scale.
Data Architecture/solution design experience.
Familiarity with Agile methodology and Jira.
Experience in data warehousing and analytical data modeling.
Knowledge of CI/CD pipelines and Git.
Experience in building reliable ETL pipelines and datasets for BI tools (Looker preferred).
Basic statistical knowledge and ability to produce high-quality technical documentation.

Build and maintain a unified and standardized data warehouse, Corpus, at SoundCloud.
Abstract the complexity of SoundCloud’s vast data ecosystem.
Collaboration with business reporting, data science, and product teams.
Gather and refine requirements, design data architecture and solutions.
Build ETL pipelines using Airflow to land data in BigQuery.
Model and build the business-ready data layer for dashboarding tools.

PythonSQLAgileETLGCPGitJiraAirflowCI/CD

Posted 2024-11-07

Apply

🔥 Senior Data Engineer (Python, Databricks)

Posted 2024-11-07

📍 Poland

🔍 Financial services industry

🏢 Company: Capco

Extensive experience with Databricks, including ETL processes and data migration.
Experience with additional cloud platforms like AWS, Azure, or GCP.
Strong knowledge of data warehousing concepts, data modeling, and SQL.
Proficiency in programming languages such as Python, SQL, and scripting languages.
Knowledge of data governance frameworks and data security principles.
Familiarity with containerization technologies such as Docker and orchestration tools like Kubernetes.
Bachelor or Master Degree in Computer Science or related field.

Design, develop, and implement robust data architecture solutions utilizing modern data platforms like Databricks.
Ensure scalable, reliable, and secure data environments that meet business requirements and support advanced analytics.
Lead the migration of data from traditional RDBMS systems to Databricks environments.
Architect and design scalable data pipelines and infrastructure to support the organization's data needs.
Develop and manage ETL processes using Databricks to ensure efficient data extraction, transformation, and loading.
Optimize ETL workflows to enhance performance and maintain data integrity.
Monitor and optimize performance of data systems to ensure reliability, scalability, and cost-effectiveness.
Collaborate with cross-functional teams to understand data requirements and deliver solutions.
Define best practices for data engineering and ensure adherence to them.
Evaluate and implement new technologies to improve data pipeline efficiency.

AWSDockerLeadershipPythonSQLETLGCPKubernetesAzureData engineeringRDBMSAnalytical Skills

Posted 2024-11-07

Apply

🔥 Lead/Senior Data Engineer

Posted 2024-11-07

📍 UK, EU

🔍 Consultancy

🏢 Company: The Dot Collective

Advanced knowledge of distributed computing with Spark.
Extensive experience with AWS data offerings such as S3, Glue, Lambda.
Ability to build CI/CD processes including Infrastructure as Code (e.g. terraform).
Expert Python and SQL skills.
Agile ways of working.

Leading a team of data engineers.
Designing and implementing cloud-native data platforms.
Owning and managing technical roadmap.
Engineering well-tested, scalable, and reliable data pipelines.

AWSPythonSQLAgileSCRUMSparkCollaborationAgile methodologies

Posted 2024-11-07

Apply

🔥 Senior Data Engineer

Posted 2024-10-23

📍 Cyprus, Malta, USA, Thailand, Indonesia, Hong Kong, Japan, Australia, Poland, Israel, Turkey, Latvia

🧭 Full-Time

🔍 Social discovery technology

🏢 Company: Social Discovery Group

3+ years of professional experience as a Data Engineer.
Confident knowledge of MS SQL including window functions, subqueries, and various joins.
Excellent knowledge of Python.
Basic query optimization skills.
Experience with Airflow.
Nice to have: experience with Google Cloud Platform (BigQuery, Storage, pub/sub).

Design, develop, and maintain SQL data warehouses, including creation and optimization of stored procedures.
Build and enhance reports using SSRS and create dynamic dashboards with Superset for actionable insights.
Develop and manage efficient data pipelines using Airflow to ensure smooth data integration and automation.

PythonSQLApache AirflowGCPAirflowData engineering

Posted 2024-10-23

Apply

🔥 Senior Data Engineer (AI), USA

Posted 2024-10-17

📍 USA, UK, Germany

💸 $70,000 - $205,000 per year

🔍 Cybersecurity

🏢 Company: Cobalt

Minimum of 5 years experience in data engineering with a strong background in Google BigQuery, Looker Studio, and DBT.
Expertise in Terraform, Python, and SQL for data transformation and infrastructure management.
Excellent verbal and written communication skills in English, enabling effective collaboration in a remote setting.
Eagerness to learn new technologies and approaches, with a proactive mindset and willingness to contribute ideas.
Understanding of Machine Learning and Generative AI.

Design, build, and maintain scalable and robust data pipelines in BigQuery, ensuring data integrity and efficiency.
Empower finance, marketing and product with data as well as providing business with valuable data insights.
Collaborate closely with Software Engineers to integrate Generative AI and Large Language Models into our data systems, focusing on automation and advanced analytics.
Manage our data lake and warehouse to also support AI and ML initiatives.
Utilize Terraform for infrastructure as code and develop Python applications for data importation, event-triggering processes, MLOps and more.
Work with various teams to understand data requirements and deliver insights and solutions that drive decision-making and product innovation.

PythonSQLCybersecurityMachine LearningData engineeringCommunication SkillsCollaborationTerraform

Posted 2024-10-17

Apply

Senior Data Engineer

Requirements:

Responsibilities:

Related Jobs

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities