Senior Data Engineer - Platform Modernization & Migration
New
I
Inallmedia.comData engineering
Location: Argentina / Brazil / Uruguay / Paraguay / Peru, Requires overlap until at least 12:00 PM Pacific Time (PT)Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- All interviews, documentation, and daily communication will be in English.
- Experience
- Minimum 2+ years of hands-on data engineering experience
- Required Skills
- PythonSQLApache AirflowSnowflakeApache KafkaSparkPySpark
Requirements
- Have at least 2 years of hands-on data engineering experience working with large-scale datasets, Apache Airflow, and Apache Spark.
- Have hands-on Python development experience and advanced SQL skills, including complex joins, CTEs, window functions, and query tuning.
- Have experience with Snowflake data modeling and schema architecture.
- Have direct experience building and testing pipelines using Apache Airflow, Apache Kafka, and Apache Spark or PySpark.
- Have expertise with Apache Iceberg and Apache Parquet data formats.
- Have experience with AWS S3 data lakes, bucket partitioning, and lifecycle policies.
- Be able to define testing strategies and independently debug complex distributed data pipelines.
- Complete the AWS WorkSpace connectivity test and submit a screenshot of the results with the application.
Responsibilities
- Execute the migration of approximately 1,000 datasets to Snowflake and upgrade orchestration workflows from Apache Airflow 2 to Airflow 3.
- Migrate legacy Hive data structures and datasets to Apache Iceberg using the Snowpark Connect framework/connectors.
- Reconfigure, update, and migrate Confluent/Apache Kafka streaming connections across cloud regions.
- Adapt existing data pipelines to Snowflake environments and optimize SQL queries using CTEs, window functions, and complex joins.
- Manage, test, and debug DAGs and workflow scheduling in Apache Airflow.
- Implement testing strategies and debug code to maintain data quality and consistency.
View Full Description & ApplyYou'll be redirected to the employer's site
