Engenheiro de Dados
New
J
JobgetherData Engineering
BrazilFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- AWSPythonSQLETLApache KafkaData engineeringSparkScala
Requirements
- Professional experience in data engineering roles focused on pipeline development and data processing.
- Strong knowledge of Apache Spark and experience with batch and streaming data processing.
- Practical experience with messaging and event-driven architectures using tools such as Apache Kafka and Amazon Kinesis.
- Proficiency in data programming languages such as Python and/or Scala.
- Advanced SQL skills and experience working with complex data queries.
- Experience handling multiple data formats, including CSV, JSON, and TXT.
- Technical ability to extract and integrate data from APIs, databases, and external platforms.
- Experience working with data quality processes, including validation, standardization, deduplication, and entity matching.
- Experience with cloud data services and storage solutions, especially within AWS environments.
Responsibilities
- Develop and maintain data ingestion pipelines, integrations, and transformation processes from multiple data sources.
- Build scalable data workflows using Apache Spark for both batch processing and real-time streaming scenarios.
- Implement and maintain event-driven data solutions using messaging and streaming technologies such as Apache Kafka and Amazon Kinesis.
- Extract and integrate raw data from different platforms, APIs, and databases.
- Design and improve transformation processes to ensure data consistency, quality, standardization, and reliability.
- Develop data preparation workflows, including validation, deduplication, and entity relationship management.
- Create and optimize data processes that support analytics and business consumption layers.
View Full Description & ApplyYou'll be redirected to the employer's site
