Applied Data Scientist, Health AI Evaluation & Datasets

New

Based in the United StatesFull-TimeSenior

Salary150,000 - 175,000 USD per year

Apply NowOpens the employer's application page

Job Details

Experience: 5+ years of experience in data science, including at least 2+ years working directly with healthcare, biomedical, clinical, payer, pharma, or life sciences data.
Required Skills: PythonSQLMachine LearningData scienceLLMGenerative AIEHRHIPAA

5+ years of experience in data science, including at least 2+ years working directly with healthcare, biomedical, clinical, payer, pharma, or life sciences data.
Strong understanding of healthcare data systems and standards, including EHR structures and clinical coding systems such as ICD-10, CPT, SNOMED CT, LOINC, and RxNorm.
Proven experience designing and building ML datasets, including annotation guidelines, sampling strategies, QA processes, and dataset validation frameworks.
Hands-on experience with LLM-based AI workflows, including evaluation design, prompt engineering, retrieval-augmented systems, and rubric-based assessment methods.
Strong programming skills in Python and SQL, with familiarity in tools such as pandas, scikit-learn, statsmodels, and modern ML/LLM ecosystems.
Solid statistical background covering sampling methods, bias and fairness analysis, inter-annotator agreement, hypothesis testing, and uncertainty estimation.
Deep understanding of healthcare privacy and compliance frameworks such as HIPAA, de-identification methodologies, and secure handling of sensitive data.
Ability to collaborate effectively with clinicians, engineers, researchers, and business stakeholders in complex, cross-functional environments.
Advanced degree in a relevant field (biostatistics, epidemiology, health informatics, data science, computer science, or related discipline) or equivalent experience.

Design, define, and operationalize high-quality healthcare datasets used for training, fine-tuning, and evaluating generative and multimodal AI systems across clinical and life sciences domains.
Translate complex healthcare objectives such as diagnosis support, medical summarization, RAG-based retrieval, and patient communication into structured dataset specifications, labeling schemas, and evaluation rubrics.
Develop clinically grounded evaluation frameworks that measure model performance across safety, accuracy, faithfulness, guideline adherence, and workflow relevance.
Design multimodal datasets spanning clinical notes, imaging, structured EHR data, claims, literature, and patient-provider communications while ensuring clinical validity and statistical rigor.
Define sampling strategies, annotation guidelines, SME review workflows, inter-annotator agreement standards, and quality assurance processes for healthcare datasets.
Build statistical and ML-based quality checks including bias analysis, subgroup performance evaluation, leakage detection, and dataset reliability metrics.
Collaborate with engineers and research scientists to integrate datasets into evaluation pipelines, including LLM-as-judge systems, benchmarking frameworks, and model comparison workflows.
Ensure data governance, compliance, and auditability across PHI-sensitive workflows, including de-identification, provenance tracking, and version control.

View Full Description & ApplyYou'll be redirected to the employer's site