Remoote.app

Work from anywhere. Start here.

JobsFree resume builderCompaniesLog in
Apply

Senior Data Engineer

Posted 2024-11-13

View full description

💎 Seniority level: Senior

📍 Location: United Kingdom

🔍 Industry: Payment and Financial Services

🏢 Company: Vitesse PSP

🪄 Skills: AWSDockerGraphQLPostgreSQLSQLAgileElasticSearchKafkaKubernetesMongoDBTableauAirflowCassandraData engineeringElasticsearchNosqlSparkCI/CDTerraformDocumentation

Requirements:
  • Experience with data pipeline orchestration tools such as Airflow, Luigi, or similar.
  • Experience with version control systems and CI/CD best practices using GitHub Actions.
  • Knowledge of data governance, privacy regulations (e.g., GDPR), and security best practices.
  • Proficiency with SQL and experience with distributed data processing tools such as Apache Spark.
  • Strong understanding of relational and NoSQL databases (e.g., PostgreSQL, MongoDB, Impala, Cassandra).
  • Experience with cloud infrastructure (Docker and Kubernetes, Terraform).
  • Experience in AWS platform architecture and cloud services.
  • A collaborative team member with Agile experience.
  • Familiarity with stream processing technologies (Kafka or Kinesis).
  • Nice to have: Experience with machine learning frameworks and pipelines, Delta Live Tables, Great Expectations, search optimizers (ElasticSearch/Lucene), REST alternatives (GraphQL, AsyncAPI), data science kits (Jupyter, Anaconda).
Responsibilities:
  • Design, build, and maintain scalable data pipelines and architectures to handle large volumes of structured and unstructured data.
  • Develop, enhance, and optimize ELT processes for ingesting, processing, and distributing data across multiple platforms in real time.
  • Build and manage data warehouses to support advanced analytics, reporting, and machine learning.
  • Implement data governance, quality checks, and validation processes to ensure the accuracy, consistency, observability, and security of data.
  • Optimize query performance and data storage costs through techniques like partitioning, indexing, vacuuming, and compression.
  • Build monitoring and alerting systems for data pipelines to proactively detect and resolve issues.
  • Optimize existing data pipelines for better performance, cost-efficiency, and scalability.
  • Work with data scientists, analysts, and business stakeholders to understand data needs.
  • Continuously research and integrate cutting-edge data technologies, tools, and practices to improve data engineering processes.
  • Team up with product engineers to identify, root cause, and resolve bugs.
  • Update documentation to help users navigate data products.
  • Ensure the data platform performs well and is always available for blue-chip clients.
Apply

Related Jobs

Apply
🔥 Senior Data Engineer
Posted 2024-11-07

📍 Canada, UK, US

🔍 Smart home technology

🏢 Company: ecobee

  • Proficiency in building data pipelines using Python and SQL.
  • Experience with Apache Spark, Apache Kafka, and Apache Airflow.
  • Experience with cloud-based data platforms, preferably GCP.
  • Familiarity with SQL-based operational databases.
  • Good understanding of machine learning lifecycle.
  • Strong experience in data modeling and schema design.
  • Experience with both batch and real-time data processing.
  • Excellent communication skills for collaborative work.

  • Design, build, and maintain scalable and efficient ETL/ELT pipelines.
  • Implement data extraction and processing solutions for analytics and machine learning.
  • Integrate diverse data sources into centralized data repositories.
  • Develop and maintain data warehousing solutions.
  • Monitor and optimize data workflows for performance and reliability.
  • Implement monitoring and logging for data pipelines.
  • Collaborate with cross-functional teams to understand data requirements.
  • Translate business requirements into technical specifications.
  • Implement data quality checks and cleansing procedures.
  • Create and maintain documentation for data pipelines.
  • Share knowledge and best practices within the team.
  • Architect data pipelines for massive IoT data streams.

LeadershipPythonSQLApache AirflowETLGCPIoTKafkaMachine LearningAirflowApache KafkaData engineeringSparkCommunication SkillsCollaboration

Posted 2024-11-07
Apply
Apply
🔥 Senior Data Engineer (Remote) - UK
Posted 2024-11-07

📍 UK

🧭 Full-Time

🔍 Knowledge management

🏢 Company: AlphaSights

  • 5+ years of hands-on data engineering development.
  • Expert in Python and SQL.
  • Experience with SQL/NoSQL databases.
  • Experienced with AWS data services.
  • Proficiency in DataOps methodologies and tools.
  • Experience with CI/CD pipelines and managing containerized applications.
  • Proficiency in workflow orchestration tools such as Apache Airflow.
  • Experience in designing, building, and maintaining Data Warehouses.
  • Collaborative experience with cross-functional teams.
  • Knowledge of ETL frameworks and best practices.

  • Design, develop, deploy and support data infrastructure, pipelines and architectures.
  • Take ownership of reporting APIs, ensuring accuracy and timeliness for stakeholders.
  • Monitor dataflows and underlying systems, promoting necessary changes for scalability and performance.
  • Collaborate directly with stakeholders to translate business problems into data-driven solutions.
  • Mentor engineers within the technical guild and support team growth.

AWSPythonSQLApache AirflowETLAirflowData engineeringNosqlCI/CD

Posted 2024-11-07
Apply
Apply
🔥 Senior Data Engineer - Data Corpus
Posted 2024-11-07

📍 US, Germany, UK

🧭 Full-Time

🔍 Music

🏢 Company: SoundCloud

  • Senior Level Data Professional with a minimum of 4 years of experience (ideal 6+ years).
  • Experience with Cloud technologies, specifically GCP (required), with AWS/Azure as a plus.
  • Experience working with BigQuery and advanced SQL knowledge.
  • Proficiency in Python and Airflow.
  • Experience with big data at terabyte/petabyte scale.
  • Data Architecture/solution design experience.
  • Familiarity with Agile methodology and Jira.
  • Experience in data warehousing and analytical data modeling.
  • Knowledge of CI/CD pipelines and Git.
  • Experience in building reliable ETL pipelines and datasets for BI tools (Looker preferred).
  • Basic statistical knowledge and ability to produce high-quality technical documentation.

  • Build and maintain a unified and standardized data warehouse, Corpus, at SoundCloud.
  • Abstract the complexity of SoundCloud’s vast data ecosystem.
  • Collaboration with business reporting, data science, and product teams.
  • Gather and refine requirements, design data architecture and solutions.
  • Build ETL pipelines using Airflow to land data in BigQuery.
  • Model and build the business-ready data layer for dashboarding tools.

PythonSQLAgileETLGCPGitJiraAirflowCI/CD

Posted 2024-11-07
Apply
Apply
🔥 Senior Data Engineer
Posted 2024-11-07

📍 Any European country

🧭 Full-Time

🔍 Software development

🏢 Company: Janea Systems

  • Proven experience as a data engineer, preferably with at least 3 or more years of relevant experience.
  • Experience designing cloud native solutions and implementations with Kubernetes.
  • Experience with Airflow or similar pipeline orchestration tools.
  • Strong Python programming skills.
  • Experience collaborating with Data Science and Engineering teams in production environments.
  • Solid understanding of SQL and relational data modeling schemas.
  • Preference for experience with Databricks or Spark.
  • Familiarity with modern data stack design and data lifecycle management.
  • Experience with distributed systems, microservices architecture, and cloud platforms like AWS, Azure, Google Cloud.
  • Excellent problem-solving skills and strong communication skills.

  • Develop and maintain data pipelines using Databricks, Airflow, or similar orchestration systems.
  • Design and implement cloud-native solutions using Kubernetes for high availability.
  • Gather product data requirements and implement solutions to ingest and process data for applications.
  • Collaborate with Data Science and Engineering teams to optimize production-ready applications.
  • Cultivate data from various sources for data scientists and maintain documentation.
  • Design modern data stack for data scientists and ML engineers.

AWSPythonSoftware DevelopmentSQLKubernetesAirflowAzureData scienceSparkCollaboration

Posted 2024-11-07
Apply
Apply
🔥 Lead/Senior Data Engineer
Posted 2024-11-07

📍 UK, EU

🔍 Consultancy

🏢 Company: The Dot Collective

  • Advanced knowledge of distributed computing with Spark.
  • Extensive experience with AWS data offerings such as S3, Glue, Lambda.
  • Ability to build CI/CD processes including Infrastructure as Code (e.g. terraform).
  • Expert Python and SQL skills.
  • Agile ways of working.

  • Leading a team of data engineers.
  • Designing and implementing cloud-native data platforms.
  • Owning and managing technical roadmap.
  • Engineering well-tested, scalable, and reliable data pipelines.

AWSPythonSQLAgileSCRUMSparkCollaborationAgile methodologies

Posted 2024-11-07
Apply
Apply
🔥 Senior Data Engineer (AI), USA
Posted 2024-10-17

📍 USA, UK, Germany

💸 $70,000 - $205,000 per year

🔍 Cybersecurity

🏢 Company: Cobalt

  • Minimum of 5 years experience in data engineering with a strong background in Google BigQuery, Looker Studio, and DBT.
  • Expertise in Terraform, Python, and SQL for data transformation and infrastructure management.
  • Excellent verbal and written communication skills in English, enabling effective collaboration in a remote setting.
  • Eagerness to learn new technologies and approaches, with a proactive mindset and willingness to contribute ideas.
  • Understanding of Machine Learning and Generative AI.

  • Design, build, and maintain scalable and robust data pipelines in BigQuery, ensuring data integrity and efficiency.
  • Empower finance, marketing and product with data as well as providing business with valuable data insights.
  • Collaborate closely with Software Engineers to integrate Generative AI and Large Language Models into our data systems, focusing on automation and advanced analytics.
  • Manage our data lake and warehouse to also support AI and ML initiatives.
  • Utilize Terraform for infrastructure as code and develop Python applications for data importation, event-triggering processes, MLOps and more.
  • Work with various teams to understand data requirements and deliver insights and solutions that drive decision-making and product innovation.

PythonSQLCybersecurityMachine LearningData engineeringCommunication SkillsCollaborationTerraform

Posted 2024-10-17
Apply
Apply
🔥 Senior Data Engineer, Data Services
Posted 2024-08-10

📍 Central EU or Americas

🧭 Full-Time

🔍 Real estate investment

🏢 Company: Roofstock👥 501-1000💰 $240.0m Series E on 2022-03-10🫂 on 2023-03-22Rental PropertyPropTechMarketplaceReal EstateFinTech

  • BS or MS in a technical field: computer science, engineering or similar.
  • 8+ years technical experience working with data.
  • 5+ years strong experience building scalable data services and applications using SQL, Python, Java/Kotlin.
  • Deep understanding of microservices architecture and RESTful API development.
  • Experience with AWS services including messaging and familiarity with real-time data processing frameworks.
  • Significant experience building and deploying data-related infrastructure and robust data pipelines.
  • Strong understanding of data architecture and related challenges.
  • Experience with complex problems and distributed systems focusing on scalability and performance.
  • Strong communication and interpersonal skills.
  • Independent worker able to collaborate with cross-functional teams.

  • Improve and maintain the data services platform.
  • Deliver high-quality data services promptly, ensuring data governance and integrity while meeting objectives and maintaining SLAs.
  • Develop effective architectures and produce key code components contributing to technical solutions.
  • Integrate a diverse network of third-party tools into a cohesive, scalable platform.
  • Continuously enhance system performance and reliability by diagnosing and resolving operational issues.
  • Ensure rigorous testing of the team's work through automated methods.
  • Support data infrastructure and collaborate with the data team on scalable data pipelines.
  • Work within an Agile/Scrum framework with cross-functional teams to deliver value.
  • Influence the enterprise data platform architecture and standards.

AWSDockerPythonSQLAgileETLSCRUMSnowflakeAirflowData engineeringgRPCRESTful APIsMicroservices

Posted 2024-08-10
Apply
Apply
🔥 Senior Data Engineer
Posted 2024-07-11

📍 United States, India, United Kingdom

🧭 Full-Time

💸 150000 - 180000 USD per year

🔍 B2B technology

  • Four-year degree in Computer Science or related field, or equivalent experience.
  • Designing frameworks and writing efficient data pipelines, including batches and real-time streams.
  • Understanding of data strategies, data analysis, and data model design.
  • Experience with the Spark Ecosystem (YARN, Executors, Livy, etc.).
  • Experience in large scale data streaming, particularly Kafka or similar technologies.
  • Experience with data orchestration frameworks, particularly Airflow or similar.
  • Experience with columnar data stores, particularly Parquet and Clickhouse.
  • Strong SDLC principles (CI/CD, Unit Testing, git, etc.).
  • General understanding of AWS EMR, EC2, S3.

  • Help build the next generation unified data platform.
  • Solve complex data warehousing problems.
  • Ensure quality, discoverability, and accessibility of data.
  • Build batch and streaming data pipelines for ingestion, normalization, and analysis.
  • Develop standard design and access patterns.
  • Lead the unification of data from multiple products.

GitAirflowClickhouseSpark

Posted 2024-07-11
Apply