Data Engineer | Senior | Threat Intelligence | Poland remote

View full description

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 2+ years of experience as a Data Engineer, working with large-scale distributed systems.
• Proven expertise in Lakehouse architecture and Apache Hudi in production environments.
• Experience with Airflow, Kafka, or streaming data pipelines.
• Strong programming skills in Python and PySpark.
• Comfortable working in a cloud-based environment (preferably AWS).

Responsibilities:

• Design, build, and manage scalable data pipelines using Python, SQL, and PySpark.
• Develop and maintain lakehouse architectures, with hands-on use of Apache Hudi for data versioning, upserts, and compaction.
• Implement efficient ETL/ELT processes for both batch and real-time data ingestion.
• Optimize data storage and query performance across large datasets (partitioning, indexing, compaction).
• Ensure data quality, governance, and lineage, integrating validation and monitoring into pipelines.
• Work with cloud-native services (preferably AWS – S3, Athena, EMR) to support modern data workflows.
• Collaborate closely with data scientists, analysts, and platform engineers to deliver reliable data infrastructure