Lead Data Engineer
New
C
CognitiveHealthcare IT
Cognitive hires only in the following designated U.S. states based on contract and business requirements: VA, DC, MD, TN, FL, AZ, CO, OR, and TX.Full-TimeLead
Salary130,000 - 160,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 8 or more years in data engineering, including 4 or more years building and operating high-volume batch ETL systems.
- Required Skills
- AWSPythonSQLApache AirflowETLSnowflakeCI/CDDatabricksPySpark
Requirements
- Bachelor's degree in Computer Science, Data Engineering, or related field.
- 8 or more years in data engineering, including 4 or more years building and operating high-volume batch ETL systems.
- Expert-level Python and PySpark skills.
- Production experience with Apache Airflow (AWS MWAA) for pipeline orchestration at scale.
- Production experience with AWS Redshift Serverless, Snowflake, and Amazon Athena.
- Hands-on experience with Databricks for large-scale data processing.
- Strong SQL skills for multi-source data transformation and reconciliation.
- Experience with AWS S3, SQS, Lambda, and Event Bridge.
- Experience with Linux environments (RHEL, CentOS, Amazon Linux 2) and Bash shell scripting.
- Experience with version control and CI/CD integration (GitHub, Jenkins, JFrog Artifactory).
- Ability to pass CMS and internal background checks for public trust.
Responsibilities
- Own summary records of prescription drug transactions (PDE) ingestion, validation, editing, and storage pipelines.
- Apply and maintain CMS-defined business rule validations and minimize manual analysis requirements.
- Engineer and maintain vendor reference data edits and provide CMS status updates on PDE submissions.
- Build, maintain, and optimize Apache Airflow and PySpark transformations across S3, Redshift Serverless, Snowflake, and Databricks.
- Monitor PDE submissions for adjustment trends and link rejected PDEs with IDR records.
- Support data transfer and new integrations for CMS and downstream stakeholders.
- Maintain target unit test coverage and track real-time PDE processing metrics.
- Operate within the CMS Lean-Agile Release Train and manage CI/CD pipelines via GitHub and Jenkins.
View Full Description & ApplyYou'll be redirected to the employer's site
