Senior Data Engineer – Platform Modernization & Migration
New
J
JobgetherData engineering
Work remotely from Brazil, Overlap with Pacific Time business hours; availability until at least 12:00 PM Pacific Time.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Strong written and verbal English skills
- Experience
- At least two years of hands-on data engineering experience
- Required Skills
- PythonSQLApache AirflowSnowflakeApache KafkaSparkPySpark
Requirements
- At least two years of hands-on data engineering experience with large-scale datasets and distributed data processing environments.
- Strong Python development skills.
- Advanced SQL expertise, including complex joins, CTEs, window functions, query optimization, and performance tuning.
- Experience with Snowflake, including data modeling, schema design, and adapting pipelines to cloud data warehouse environments.
- Practical experience developing and testing pipelines with Apache Airflow and Apache Spark or PySpark.
- Familiarity with Apache Kafka and streaming data architectures.
- Experience with Apache Iceberg, Apache Parquet, AWS S3, and data lake concepts, including partitioning and storage lifecycle policies.
- Ability to design testing approaches, troubleshoot distributed systems, and independently resolve data pipeline issues.
- Strong written and verbal English skills for technical discussions, documentation, and collaboration with international teams.
- Ability to work independently, prioritize tasks, and overlap with the core engineering team until at least 12:00 PM Pacific Time.
- Ability to access a client-provisioned AWS WorkSpace with reliable connectivity and submit a screenshot of the required connectivity test results with the application.
- Availability for a full-time, permanent position and willingness to collaborate across time zones.
Responsibilities
- Execute the end-to-end migration of approximately 1,000 datasets to Snowflake using established migration blueprints.
- Upgrade orchestration workflows from Apache Airflow 2 to Airflow 3 by adapting, testing, and debugging DAGs.
- Migrate legacy Hive-based data structures and datasets to Apache Iceberg, using Snowpark Connect frameworks and connectors where applicable.
- Reconfigure and migrate Confluent and Apache Kafka streaming connections across cloud regions.
- Adapt data pipelines for Snowflake environments and optimize SQL queries using CTEs, window functions, and complex joins.
- Build, maintain, test, and troubleshoot workflows with Python, Apache Spark, PySpark, Apache Airflow, and Apache Kafka.
- Develop and execute testing strategies and validate data accuracy, consistency, and integrity throughout migrations.
- Identify bottlenecks and optimize distributed pipelines, storage structures, and query execution.
- Support AWS S3 data lake operations, including storage organization, bucket partitioning, and lifecycle management.
- Coordinate migration plans, resolve technical challenges, and communicate progress, risks, and findings to distributed stakeholders.
View Full Description & ApplyYou'll be redirected to the employer's site
