Big Data Engineer (Remote in Romania)

View full description

Requirements:

• 8+ years of IT experience with at least 4 years working with Azure Databricks.
• Strong proficiency in Python for data engineering, along with expertise in PySpark and SQL.
• Knowledge of Azure Data components including Data Factory, Data Lake, SQL Data Warehouse (DW), and Azure SQL.
• Experience in data modeling, source system analysis, and developing technical designs for data flows.
• Familiarity with data profiling, cataloging, and mapping processes.
• Experience with data visualization and exploration tools.

Responsibilities:

• Lead the technical planning and execution of data migration, including ingestion, transformation, and storage within Azure Data Factory and Azure Data Lake.
• Design and implement scalable data pipelines for seamless data movement from various sources using Azure Databricks.
• Develop reusable frameworks for ingesting large datasets and implement data validation and cleansing mechanisms.
• Work with real-time streaming technologies to process and ingest data effectively.
• Provide technical support during and after migration, resolving challenges as they arise.
• Stay up-to-date on advancements in cloud computing and data engineering, recommending best practices and industry standards.