Data Engineer Lead

View full description

Requirements:

• Expertise in ETL tools like Informatica PowerCenter, Informatica Cloud ( IICS )
• Extensive experience in data engineering, with a focus on Azure cloud platform.
• Proficiency in Azure services like Azure Data Factory, Azure Databricks, Azure SQL Data Warehouse, and Azure Stream Analytics.
• Strong programming skills in languages such as Python, SQL, or Scala for data manipulation and transformation.
• Experience with big data technologies like Hadoop, Spark, or Hive is a plus.
• Familiarity with data visualization tools like Power BI or Tableau.
• Knowledge of data warehousing concepts, data lakes, and real-time data processing.

Responsibilities:

• Design and Implement robust ETL workflows using Informatica to extract, transform and load data from diverse sources.
• Design data solutions, including data lakes, data warehouses, and real-time data processing systems, leveraging Azure services like Azure Data Lake Storage, Azure Synapse Analytics
• Lead a team of azure data engineers in designing, building, and maintaining scalable and efficient data pipelines on Azure cloud platform, utilizing services like Azure Synapse, Azure Data Factory, Azure Databricks, Azure SQL Database, and others.
• Integrate diverse data sources into Azure-based solutions.
• Ensure smooth and efficient ETL processes, real-time data ingestion, and data transformation.
• Implement data integration best practices.
• Design and implement data models and schemas to support business requirements.
• Ensure data accuracy, consistency, and reliability.
• Optimize data structures for performance and scalability.
• Implement robust data security measures, including encryption, access control, and data masking.
• Ensure compliance with data privacy regulations and company policies.
• Monitor and optimize data pipelines and queries for performance and efficiency.
• Implement caching, partitioning, and indexing strategies.
• Troubleshoot and resolve performance issues.
• Establish and enforce data quality standards.
• Implement data governance practices, metadata management, and data lineage tracking.
• Ensure data quality through validation and cleansing processes.
• Collaborate with business stakeholders to understand data requirements and deliver.
• Create and maintain comprehensive technical documentation, including system architecture, design documents, and deployment procedures.
• Ensure knowledge sharing within the team.
• Implement Lean daily management and Lean continuous improvement concepts in Application development and operations.