Senior Data Consultant

View full description

Requirements:

• 8-10 years of experience in data engineering or a related role with a strong background in building and optimizing data pipelines.
• Expertise in PySpark for large-scale data processing and distributed computing.
• Strong experience with Azure data technologies (Azure Data Factory, Azure Databricks, Azure SQL Database, etc.).
• Proficiency in SQL and experience working with relational and NoSQL databases.
• Experience with cloud-based data platforms and services, especially in Azure.
• Solid understanding of ETL processes and data modeling techniques.
• Strong problem-solving and troubleshooting skills.
• Experience working with version control systems (e.g., Git) and continuous integration/continuous deployment (CI/CD) practices.
• Strong communication skills and the ability to work effectively within a collaborative, small-team environment.
• A proactive, self-starter attitude with a passion for data engineering and technology.

Responsibilities:

• Design, build, and maintain scalable data pipelines for the collection, transformation, and storage of data.
• Work with large, complex datasets to ensure efficient processing and integration.
• Develop data engineering solutions using Azure technologies and PySpark for distributed data processing.
• Implement ETL processes and automate data workflows to support analytics, reporting, and business intelligence initiatives.
• Collaborate with data scientists, business analysts, and stakeholders to understand data requirements and deliver actionable insights.
• Ensure data quality, consistency, and reliability across all pipelines and datasets.
• Optimize data models, storage, and processing workflows for performance and scalability.
• Contribute to the architecture and design decisions for cloud-based data platforms.
• Mentor junior team members and foster a collaborative and innovative team culture.
• Troubleshoot and resolve issues related to data pipelines, ensuring high availability and performance.