Senior Data Engineer
New
S
SoftetaTelecommunications data
Workable workplace: remote; Workable locations: Kraków, Lesser Poland Voivodeship, PolandFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonAirflowApache KafkaSparkdbt
Requirements
- Bring deep hands-on experience with Airflow, Spark, dbt, BigQuery, and Python.
- Have a proven track record leading a live pipeline migration technically from conception to completion.
- Have hands-on experience refactoring production pipelines while verifying identical output.
- Demonstrate success establishing engineering best practices, including testing, code reviews, and incident resolution, within a team.
- Bring concrete mentoring or technical leadership experience.
- Nice to have: experience with encryption at rest, data retention policies, or regulatory compliance such as GDPR.
- Nice to have: experience with envelope encryption using key management services such as GCP KMS, BigQuery AEAD functions, or Tink.
- Nice to have: experience with BigQuery column-level security, policy tags, and Google Cloud Data Catalog.
- Nice to have: experience with event-driven architectures using Apache Kafka.
Responsibilities
- Propose and lead architecture for user-data workflows across Airflow, Spark, dbt, and BigQuery, including encryption points, derived tables, backfills, and partitions.
- Sequentially migrate existing pipelines using a custom regression harness to guarantee identical output.
- Establish code reviews, testing protocols, incident post-mortems, and release checklists.
- Implement verification jobs for end-to-end data lifecycle rules.
- Maintain pipelines and participate in incident response and root-cause resolution.
- Document design choices using Architecture Decision Records (ADRs).
View Full Description & ApplyYou'll be redirected to the employer's site
