Data Engineer
S
Simple Technology SolutionsInformation Technology
Location: Remote, 8am-5pm Eastern TimeFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 3-5 years
- Required Skills
- AWSPostgreSQLPythonSQLApache AirflowCloudFormationPySpark
Requirements
- Bachelor's degree in Computer Science, Information Systems, Data Engineering, or related field.
- 3-5 years of experience in data engineering or related role.
- Hands-on experience with Python (PEP 8), PySpark, and SQL.
- Experience with AWS services including Glue, S3, MWAA (Airflow), Lambda, SNS, and SQS.
- Familiarity with Apache Iceberg, Parquet, and ORC file formats.
- Experience with PostgreSQL.
- Familiarity with Trino or Athena.
- Experience with CloudFormation, GitHub workflows, and CI/CD.
- Experience with automated testing frameworks and writing unit/integration tests.
- Experience in agile development environments.
Responsibilities
- Develop new ETL pipelines and data ingestion processes using AWS Glue, PySpark, Airflow, Lambda, and SNS.
- Ingest structured and semi-structured datasets into S3 landing, raw, and curated zones using Apache Iceberg.
- Configure static ETL metadata in a centralized PostgreSQL metadata store.
- Monitor production jobs and participate in operations support rotations.
- Ensure ETL Load Reports and Gap Reports are maintained.
- Build and maintain materialized views and semantic layer objects in Trino and Athena.
- Produce and maintain technical documentation including ETL designs, data models, and mapping documents.
- Write unit and integration tests achieving 90% minimum code coverage.
- Deploy ETL resources using CloudFormation through the agency CICD pipeline.
View Full Description & ApplyYou'll be redirected to the employer's site
