Senior Data Engineer
New
C
CI&TData engineering
Listing location: Brazil; Workplace type: RemoteFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- AWSPythonAWS LambdaPySpark
Requirements
- Strong knowledge of Python.
- Practical experience with PySpark.
- Experience with AWS, including S3, Glue, EC2, Lambda, and Athena.
- Knowledge of relational databases.
- Experience tuning and improving data-pipeline performance.
- Experience automating unit tests and applying the test-pyramid strategy.
- Ability to refine technical stories and solutions.
- Knowledge of code quality and good programming practices.
- Ability to coordinate with the team and stakeholders.
- Knowledge of cloud cost optimization and FinOps practices for data pipelines.
Responsibilities
- Develop and evolve data-ingestion applications using AWS serverless tools and ECS-based architectures.
- Understand technical needs and participate in inception and refinement sessions.
- Write code and structure automated tests to assure the quality of deliverables.
- Conduct code reviews and deploy to production.
- Manage incidents.
- Optimize data jobs and pipelines through partitioning, skew balancing, and performance tuning.
- Design and maintain pipeline orchestration using AWS Step Functions and AWS EventBridge.
- Configure and monitor observability using AWS CloudWatch and, optionally, Datadog.
- Apply FinOps practices to use cloud resources efficiently and identify cost-optimization opportunities.
View Full Description & ApplyYou'll be redirected to the employer's site
