Data Engineer, Azure
New
B
BluelightSoftware Consultancy
Latin AmericaFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonSQLETLGitMicrosoft SQL ServerAzure DevOpsPySpark
Requirements
- Bachelor’s degree in Computer Science, IT, or related field, or equivalent experience.
- Proven experience in ETL data engineering with significant expertise in Python (PySpark).
- Proficiency in Azure Synapse Analytics resources, including Notebooks, Pipelines, Linked Services, and Azure Key Vault.
- Demonstrated ability to write complex SQL queries, optimize performance, and work with SparkSQL and MS SQL.
- Experience with version control systems, specifically Git (Azure DevOps).
- Knowledge of data integration best practices.
- Strong problem-solving and analytical skills with attention to detail.
- Excellent communication and collaboration skills for working in team environments.
Responsibilities
- Develop and maintain ETL data engineering processes using Python (PySpark) within Azure Synapse Analytics Notebooks and/or Pipelines.
- Apply data warehousing expertise to design and build effective data storage structures in a Massively Parallel Processing (MPP) SQL Pool.
- Extract data from various sources, including REST APIs, SQL database tables, and CSV files.
- Contribute to the implementation of Data Fabric concepts including data lakes, lakehouses, delta lakes, and data cataloging.
- Collaborate with data architects on data modeling and schemas to align with business requirements.
- Implement data quality checks and validation processes to ensure data accuracy.
- Identify and resolve performance bottlenecks in ETL notebooks and pipelines.
- Monitor ETL jobs and implement solutions for pipeline reliability.
View Full Description & ApplyYou'll be redirected to the employer's site
