Senior Data Engineer
New
J
JobgetherLegal technology
Based in BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonSQLApache AirflowSparkData modelingPySpark
Requirements
- Have solid professional experience in Data Engineering and data solution architecture.
- Demonstrate strong proficiency in SQL and Python for data manipulation, transformation, and processing.
- Have hands-on experience developing and maintaining ETL/ELT pipelines.
- Know Lakehouse architecture and the Medallion data model (Bronze, Silver, and Gold).
- Have experience with Big Data platforms and distributed processing technologies such as Apache Spark and PySpark.
- Be familiar with cloud-based data storage and processing platforms.
- Have experience integrating and processing structured and unstructured data.
- Know data modeling and the development of Data Lakes and Data Warehouses.
- Have experience with pipeline orchestration tools such as Apache Airflow or similar technologies.
- Understand data governance, quality management, lineage, and security best practices.
- Be familiar with preparing and optimizing datasets for machine learning models and LLMs.
- Generative AI, RAG, document processing, OCR, vector databases, embedding techniques, legal or banking datasets, cloud AI services, and DataOps/CI/CD experience are listed as nice to have.
Responsibilities
- Design, develop, and maintain scalable data pipelines for analytics and AI initiatives.
- Manage ingestion, extraction, transformation, cleaning, standardization, and enrichment of large datasets.
- Design and implement Lakehouse architectures and Bronze, Silver, and Gold Medallion layers.
- Transform raw data into structured, reliable, reusable data products, with emphasis on the Gold layer.
- Prepare high-quality datasets for AI solutions, machine learning models, and LLMs.
- Develop ETL/ELT processes that integrate structured and unstructured data from multiple sources.
- Implement data validation, quality assurance, lineage, traceability, and monitoring.
- Process legal, judicial, contractual, and documentary datasets, including extracting information from unstructured documents.
- Collaborate with Data Scientists, AI professionals, Technology teams, and Legal specialists on data integration and delivery.
- Ensure data security, governance, access control, and LGPD compliance, and improve infrastructure performance, cost, and scalability.
View Full Description & ApplyYou'll be redirected to the employer's site
