- Design and implement batch and streaming ingestion from APIs, relational databases, file drops, event streams, and external partners.
- Build, test, and optimize Python and PySpark ETL/ELT pipelines and curated datasets for analytics, reporting, visualization, and machine learning.
- Build and operate an AWS-native lakehouse on Amazon S3 using Apache Iceberg and open columnar formats such as Apache Parquet.
- Enable querying with AWS services such as Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift.
- Implement metadata management, data lineage, governance, and fine-grained access controls using AWS-native services.
- Build data quality checks and publish measurable SLAs/SLOs.
- Automate AWS provisioning and CI/CD for data pipelines and lakehouse components across environments.
- Implement observability, operational runbooks, and incident-response procedures, and improve platform performance, reliability, security, and cost.
AWSPythonSQL+2 more