Lead Data Engineer

New

Smart WorkingAI data infrastructure

United KingdomContractLead

Salary not disclosed

Apply NowOpens the employer's application page

Job Details

Experience: 7+ years of professional experience, with the majority of that experience in dedicated data engineering roles.
Required Skills: PostgreSQLPythonApache AirflowKafkaData engineeringSparkData modelingDistributed Systems

Have 7+ years of professional experience, with most experience in dedicated data engineering roles.
Have strong experience designing and building data pipelines and distributed data systems.
Have experience with relational databases; PostgreSQL is preferred, with MySQL or similar acceptable.
Have experience working with NoSQL databases.
Have strong programming experience in Python.
Demonstrate the ability to make and justify architectural decisions.
Have experience building scalable backend systems.
Have experience designing data models and storage architectures.
Have a strong understanding of data-processing performance and optimization.
Experience with Apache Spark, Apache Airflow, Kafka, and Elasticsearch or OpenSearch is highly desirable.
Experience with PostgreSQL, MongoDB, and vector databases such as Qdrant, Milvus or pgvector is highly desirable.
Experience with Python data-processing libraries such as Pandas or Polars is highly desirable.
Experience with AI or machine-learning platforms, stream processing, event-driven architectures, or cloud infrastructure such as GCP, AWS or Azure is a plus.

Architect and build scalable data pipelines and infrastructure for AI and product systems.
Design and maintain data ingestion, transformation and storage architectures for operational and AI workloads.
Develop and manage batch and real-time data pipelines.
Build and optimize systems for vector search, retrieval and machine-learning data pipelines.
Ensure data reliability, security and governance across the platform.
Collaborate with AI and backend engineering teams on training, inference and product features.
Implement monitoring, observability and data quality frameworks.
Optimize large-scale datasets and query systems, and contribute to architecture decisions and long-term data strategy.
Define standards and hiring expectations for the data function as its founding data hire.
Partner with founders and product leadership to inform product decisions using data capabilities.

View Full Description & ApplyYou'll be redirected to the employer's site