Data Engineer (Apache Spark)

View full description

Requirements:

• At least 3 years of professional experience as a data engineer.
• Undergraduate or graduate degree in Computer Science, Engineering, Mathematics, or similar.
• Excellent command in spoken and written English, at least C1.
• Strong professional experience with Apache Spark.
• Hands-on experience managing production spark clusters in Databricks.
• Experience in CI/CD of data jobs in Spark.
• Great analytical skills, attention to detail, and creative problem-solving skills.
• Great customer service and troubleshooting skills.

Responsibilities:

• Design and manage batch data pipelines, including file ingestion, transformation, and Delta Lake/table management.
• Implement scalable architectures for batch and streaming workflows.
• Leverage Microsoft equivalents of BigQuery for efficient querying and data storage.