Data Engineer

Posted 2024-11-07

View full description

📍 Location: United Kingdom, England

🔍 Industry: Consultancy

🏢 Company: The Dot Collective

🪄 Skills: PythonSQLAgileSCRUMSparkCollaboration

Requirements:

Good knowledge of distributed computing with Spark.
Understanding of cloud architecture principles and best practices.
Hands-on experience in designing, deploying, and managing cloud resources.
Excellent Python and SQL skills.
Experience in cloud automation and orchestration using tools such as CloudFormation or Terraform.
Agile ways of working.

Responsibilities:

Implement cloud-native data platforms.
Engineer scalable and reliable data pipelines.
Monitor and perform tuning of cloud-based applications and services.

Apply

Related Jobs

Apply

🔥 Data Engineer

Posted 2024-11-20

📍 Argentina, Spain, England, United Kingdom, Lisbon, Portugal

🧭 Full-Time

🔍 Web3

🏢 Company: Reown

5+ years working in the analytics stack within a fast-paced environment.
3+ years production experience with SQL templating engines like DBT.
Experience with distributed query engines (Bigquery, Athena, Spark), data warehouses, and BI tools.
Strong understanding of software engineering principles, coding standards, design patterns, version control (e.g., Git), testing methodologies, and CI/CD processes.
Experience with AWS/GCP/Azure services for deployment and management.
Familiarity with GitHub, CI/CD pipelines, GitHub Actions, and Terraform.
Ability to write Python scripts for ETL processes and data manipulation.
Proficient with libraries like pandas for analysis and transformation.
Experience handling various data formats (e.g., CSV, JSON, Parquet).
Strong problem-solving skills and communication abilities to discuss technical concepts.

Write complex SQL queries that extract and combine data from on-chain and off-chain logs for analytics.
Create dashboards and tools for team data discoverability and KR tracking.
Perform deep-dive analyses into specific topics for internal stakeholders.
Help design, implement, and evolve Reown's on-chain data infrastructure.
Build, maintain, and monitor end-to-end data pipelines for new datasets and features.
Write health-checks and alerts to ensure data correctness, consistency, and freshness.
Meet with product managers and stakeholders to understand data needs and detect new product opportunities.

AWSPythonSQLData AnalysisDesign PatternsETLGCPGitTableauAzureClickhouseData analysisPandasSparkCommunication SkillsCI/CDTerraformWritten communication

Posted 2024-11-20

Apply

🔥 Data Engineer, Blockchain

Posted 2024-11-15

📍 North America, South America, Europe

💸 100000 - 500000 USD per year

🔍 Web3, blockchain

🏢 Company: Edge & Node

A self-motivated, team member with keen attention to detail.
Proactive collaboration with team members and a willingness to adapt to a growing environment.
Familiarity and experience with Rust, particularly focusing on data transformation and ingestion.
A strong understanding of blockchain data structures and ingestion interfaces.
Experience in real-time data handling, including knowledge of reorg handling.
Familiarity with blockchain clients like Geth and Reth is a plus.
Adaptability to a dynamic and fully-remote work environment.
Rigorous approach to software development that reflects a commitment to excellence.

Develop and maintain data ingestion adapters for various blockchain networks and web3 protocols.
Implement data ingestion strategies for both historical and recent data.
Apply strategies for handling block reorgs.
Optimize the latency of block ingestion at the chain head.
Write interfaces with file storage protocols such as IPFS and Arweave.
Collaborate with upstream data sources, such as chain clients and tracing frameworks, and monitor the latest upstream developments.
Perform data quality checks, cross-checking data across multiple sources and investigating any discrepancies that arise.

Software DevelopmentBlockchainData StructuresRustCollaborationAttention to detail

Posted 2024-11-15

Apply

🔥 Network Data Engineer

Posted 2024-11-14

📍 United Kingdom

🧭 Contract

🏢 Company: Axiom Software Solutions Limited

Expertise in L2/L3 IP protocols such as HSRP, OSPF, BGP, MPLS, and VRF.
Deep knowledge of configuration and troubleshooting of IP protocols.
Experience in multi-vendor environments specialized in Viptela SD-WAN, Cisco SD-Access.
CCNP-ENCOR or CCIE Enterprise Infrastructure certified.
Good understanding of Data Center infrastructure and connectivity.
Knowledge of Vulnerability Management and Lifecycle Management.

Extensive support for DC Operation support.
Troubleshooting expert of network-related issues.
Interaction with customers to understand network and service delivery requirements.
Handling deployment and implementation of network solutions.
Preparing high-level and low-level design documents.
Providing consultation on network design with SD-WAN and SD-Access.

CiscoProject CoordinationCommunication SkillsCollaborationProblem SolvingAttention to detailOrganizational skillsPresentation skillsTime ManagementWritten communicationDocumentation

Posted 2024-11-14

Apply

🔥 Senior Data Engineer

Posted 2024-11-13

📍 United Kingdom

🔍 Payment and Financial Services

🏢 Company: Vitesse PSP

Experience with data pipeline orchestration tools such as Airflow, Luigi, or similar.
Experience with version control systems and CI/CD best practices using GitHub Actions.
Knowledge of data governance, privacy regulations (e.g., GDPR), and security best practices.
Proficiency with SQL and experience with distributed data processing tools such as Apache Spark.
Strong understanding of relational and NoSQL databases (e.g., PostgreSQL, MongoDB, Impala, Cassandra).
Experience with cloud infrastructure (Docker and Kubernetes, Terraform).
Experience in AWS platform architecture and cloud services.
A collaborative team member with Agile experience.
Familiarity with stream processing technologies (Kafka or Kinesis).
Nice to have: Experience with machine learning frameworks and pipelines, Delta Live Tables, Great Expectations, search optimizers (ElasticSearch/Lucene), REST alternatives (GraphQL, AsyncAPI), data science kits (Jupyter, Anaconda).

Design, build, and maintain scalable data pipelines and architectures to handle large volumes of structured and unstructured data.
Develop, enhance, and optimize ELT processes for ingesting, processing, and distributing data across multiple platforms in real time.
Build and manage data warehouses to support advanced analytics, reporting, and machine learning.
Implement data governance, quality checks, and validation processes to ensure the accuracy, consistency, observability, and security of data.
Optimize query performance and data storage costs through techniques like partitioning, indexing, vacuuming, and compression.
Build monitoring and alerting systems for data pipelines to proactively detect and resolve issues.
Optimize existing data pipelines for better performance, cost-efficiency, and scalability.
Work with data scientists, analysts, and business stakeholders to understand data needs.
Continuously research and integrate cutting-edge data technologies, tools, and practices to improve data engineering processes.
Team up with product engineers to identify, root cause, and resolve bugs.
Update documentation to help users navigate data products.
Ensure the data platform performs well and is always available for blue-chip clients.

AWSDockerGraphQLPostgreSQLSQLAgileElasticSearchKafkaKubernetesMongoDBTableauAirflowCassandraData engineeringElasticsearchNosqlSparkCI/CDTerraformDocumentation

Posted 2024-11-13

Apply

🔥 Lead Data Engineer

Posted 2024-11-07

📍 North America, Latin America, Europe

🔍 Data consulting

Bachelor’s degree in engineering, computer science or equivalent area.
5+ years in related technical roles such as data management, database development, and ETL.
Expertise in evaluating and integrating data ingestion technologies.
Experience in designing and developing data warehouses with various platforms.
Proficiency in building ETL/ELT ingestion pipelines with tools like DataStage or Informatica.
Cloud experience on AWS; Azure and GCP experience is a plus.
Proficiency in Python scripting; Scala is required.

Designing and developing Snowflake Data Cloud solutions.
Creating data ingestion pipelines and working on data architecture.
Ensuring data governance and security throughout customer projects.
Leading technical teams and collaborating with clients on data initiatives.

AWSLeadershipPythonSQLAgileETLOracleSnowflakeData engineeringSparkCollaboration

Posted 2024-11-07

Apply

🔥 Staff Data Engineer

Posted 2024-11-07

📍 Canada, United States, United Kingdom

🔍 Smart home technology

🏢 Company: ecobee

10+ years of experience in data/software engineering with proven track record.
Extensive experience in building and maintaining scalable data pipelines with tools like Apache Spark, Kafka, and Airflow.
Expertise in cloud data platforms (AWS, GCP, or Azure), focusing on distributed systems.
Solid understanding of end-to-end data systems and machine learning deployment.
Knowledge in data security, governance, and compliance.
Experience in data architecture and engineering methodologies across industries.
Experience with real-time data processing and analytics platforms.
Proven ability to mentor and guide engineers across teams.

Lead the design and implementation of scalable data pipelines and systems for complex problems.
Contribute to ecobee’s system architecture with impactful designs.
Take end-to-end ownership of components within your domain.
Define and track SLAs for components to ensure reliability.
Mentor engineers and drive knowledge-sharing initiatives.
Collaborate across squads to align technical discussions with strategic goals.
Anticipate future data challenges and propose strategies.
Evaluate and recommend new technologies for data engineering.

AWSLeadershipGCPIoTKafkaMachine LearningStrategyAirflowAzureData engineeringSparkCollaboration

Posted 2024-11-07

Apply

🔥 Senior Data Engineer

Posted 2024-11-07

📍 Canada, UK, US

🔍 Smart home technology

🏢 Company: ecobee

Proficiency in building data pipelines using Python and SQL.
Experience with Apache Spark, Apache Kafka, and Apache Airflow.
Experience with cloud-based data platforms, preferably GCP.
Familiarity with SQL-based operational databases.
Good understanding of machine learning lifecycle.
Strong experience in data modeling and schema design.
Experience with both batch and real-time data processing.
Excellent communication skills for collaborative work.

Design, build, and maintain scalable and efficient ETL/ELT pipelines.
Implement data extraction and processing solutions for analytics and machine learning.
Integrate diverse data sources into centralized data repositories.
Develop and maintain data warehousing solutions.
Monitor and optimize data workflows for performance and reliability.
Implement monitoring and logging for data pipelines.
Collaborate with cross-functional teams to understand data requirements.
Translate business requirements into technical specifications.
Implement data quality checks and cleansing procedures.
Create and maintain documentation for data pipelines.
Share knowledge and best practices within the team.
Architect data pipelines for massive IoT data streams.

LeadershipPythonSQLApache AirflowETLGCPIoTKafkaMachine LearningAirflowApache KafkaData engineeringSparkCommunication SkillsCollaboration

Posted 2024-11-07

Apply

🔥 Senior Data Engineer (Remote) - UK

Posted 2024-11-07

📍 UK

🧭 Full-Time

🔍 Knowledge management

🏢 Company: AlphaSights

5+ years of hands-on data engineering development.
Expert in Python and SQL.
Experience with SQL/NoSQL databases.
Experienced with AWS data services.
Proficiency in DataOps methodologies and tools.
Experience with CI/CD pipelines and managing containerized applications.
Proficiency in workflow orchestration tools such as Apache Airflow.
Experience in designing, building, and maintaining Data Warehouses.
Collaborative experience with cross-functional teams.
Knowledge of ETL frameworks and best practices.

Design, develop, deploy and support data infrastructure, pipelines and architectures.
Take ownership of reporting APIs, ensuring accuracy and timeliness for stakeholders.
Monitor dataflows and underlying systems, promoting necessary changes for scalability and performance.
Collaborate directly with stakeholders to translate business problems into data-driven solutions.
Mentor engineers within the technical guild and support team growth.

AWSPythonSQLApache AirflowETLAirflowData engineeringNosqlCI/CD

Posted 2024-11-07

Apply

🔥 Senior Data Engineer - Data Corpus

Posted 2024-11-07

📍 US, Germany, UK

🧭 Full-Time

🔍 Music

🏢 Company: SoundCloud

Senior Level Data Professional with a minimum of 4 years of experience (ideal 6+ years).
Experience with Cloud technologies, specifically GCP (required), with AWS/Azure as a plus.
Experience working with BigQuery and advanced SQL knowledge.
Proficiency in Python and Airflow.
Experience with big data at terabyte/petabyte scale.
Data Architecture/solution design experience.
Familiarity with Agile methodology and Jira.
Experience in data warehousing and analytical data modeling.
Knowledge of CI/CD pipelines and Git.
Experience in building reliable ETL pipelines and datasets for BI tools (Looker preferred).
Basic statistical knowledge and ability to produce high-quality technical documentation.

Build and maintain a unified and standardized data warehouse, Corpus, at SoundCloud.
Abstract the complexity of SoundCloud’s vast data ecosystem.
Collaboration with business reporting, data science, and product teams.
Gather and refine requirements, design data architecture and solutions.
Build ETL pipelines using Airflow to land data in BigQuery.
Model and build the business-ready data layer for dashboarding tools.

PythonSQLAgileETLGCPGitJiraAirflowCI/CD

Posted 2024-11-07

Apply

🔥 Data Engineer - OpenData Commercial

Posted 2024-11-07

📍 UK

🔍 Life sciences

Must be based in the UK.
Hold legal work authorization.
Specific skill requirements are not provided in the job description.

Drive impactful change by harmonizing data operations globally.
Enhance data quality.
Create the tools and processes necessary to efficiently store, manage, and compile data.
Contribute directly to the success of OpenData.

PostgreSQLPythonSoftware DevelopmentSQLGitData engineeringPostgresCommunication SkillsAnalytical SkillsCollaboration

Posted 2024-11-07

Apply

Data Engineer

Requirements:

Responsibilities:

Related Jobs

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities

🔧 Requirements

💡 Responsibilities