Senior Data Engineer
New
T
Trust WalletCryptocurrency, Web3
Global and fully remote roleFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 3+ years
- Required Skills
- AWSPythonSQLData engineeringSparkdbtDatabricks
Requirements
- 3+ years as a Data Engineer, with hands-on ownership of production pipelines and lakehouse or data warehouse architecture.
- Strong practical experience with Databricks, Delta Lake and Spark, including both streaming and batch processing.
- Solid understanding of dimensional modelling, slowly changing dimensions and data warehouse design.
- Experience with dbt or an equivalent framework for managing transformations, testing and lineage.
- Strong cloud fundamentals, including identity and access management, object storage, networking and infrastructure as code (AWS preferred).
- Proficiency in Python and SQL for writing production transformations, services, and connectors.
- Experience implementing testing, monitoring and governance practices in cloud data environments.
Responsibilities
- Architect and maintain robust, scalable and secure data infrastructure on Databricks, covering both streaming and batch workloads.
- Design, develop and maintain data pipelines, primarily in Python and Spark, to automate ingestion and transformation across internal systems, external providers and on-chain sources.
- Design and maintain dimensional data models and transformation layers in dbt, with tests and documented contracts.
- Oversee the data lake and lakehouse layers, ensuring efficient storage, effective partitioning, and high data quality.
- Integrate Databricks with a wide range of data sources, including change data capture from operational databases, third-party APIs and blockchain data.
- Optimise pipelines and storage for performance, reliability and cost efficiency at scale.
- Apply best practices for governance, security and compliance in cloud and Databricks environments.
- Work closely with platform engineers, data analysts and other stakeholders to understand data requirements and document infrastructure.
View Full Description & ApplyYou'll be redirected to the employer's site
