Senior Data Engineer & Agent Platform Engineer (GCP)
New
J
JobgetherHealthcare AI
Based in BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonSQLTerraformBigQuery
Requirements
- Strong proficiency in Python and SQL.
- Hands-on experience with GCP data engineering services, particularly BigQuery, Cloud Storage, Dataflow or Dataproc, and Pub/Sub.
- Experience designing and orchestrating data pipelines using Cloud Composer/Airflow, Cloud Workflows, or similar tools.
- Knowledge of GCP data governance and security services, including Cloud IAM, Dataplex, policy tags, and Cloud KMS.
- Practical experience with Terraform and CI/CD tools such as Cloud Build or GitHub Actions.
- Experience working with GCP environments spanning multiple projects, organizations, and resource folders.
- Strong understanding of distributed and event-driven architectures, including messaging, retries, idempotency, and data consistency.
- Practical experience preparing and managing data for at least one production-level LLM application, such as RAG or AI agents with tool-calling capabilities.
- Knowledge of LGPD requirements for sensitive data processing, including anonymization, masking, and privacy protection.
- Ability to communicate technical trade-offs involving data access, batch versus streaming architectures, cost, and latency.
- Strong documentation habits and ability to establish clear ownership and responsibilities across technical teams.
Responsibilities
- Design, develop, and maintain batch and streaming data ingestion, transformation, and quality pipelines using GCP services.
- Build scalable Data Lake and Lakehouse architectures with BigQuery and BigLake, including data layers, partitioning, and performance and cost optimizations.
- Define and maintain data contracts, freshness requirements, SLAs, data quality, consistency, and lineage.
- Prepare datasets for AI applications through cleaning, anonymization, chunking, and embedding generation.
- Implement LGPD safeguards for sensitive healthcare information, including masking, anonymization, policy tags, and data residency controls.
- Design secure cross-project data access using Cloud IAM, Dataplex, BigLake, and Analytics Hub.
- Enable internal systems and data sources to be used by AI agents through MCP servers and MCP Toolbox, coordinating integrations with AI teams.
- Develop MLOps and LLMOps pipelines with evaluations, security guardrails, and monitoring for data drift and response quality.
- Provision data and AI infrastructure with Terraform and CI/CD, and monitor costs using BigQuery billing exports and resource labels.
- Establish observability and contribute to resilience and disaster recovery for data and AI services.
View Full Description & ApplyYou'll be redirected to the employer's site
