Model Test and Measurement Engineer
New
D
DEFCON AIAI analytics
Remote, USAFull-TimeSenior
Salary$150,000–$190,000
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years with model validation as a named responsibility, not a side task
- Required Skills
- PythonSQL
Requirements
- Have 5+ years of experience with model validation as a named responsibility, not a side task.
- Have direct experience with ground-truth construction and sampling design.
- Have strong Python skills.
- Have strong SQL skills.
- Be able to work independently from the teams whose models you evaluate.
- Be a US citizen.
- Hold an active US Secret clearance.
- Regulated-industry or government model-risk background is preferred.
- Experience with fairness or disparate-effect testing and human-oversight documentation is preferred.
- NIST AI RMF or comparable practice is preferred.
- An active Top Secret clearance is preferred.
Responsibilities
- Construct labeled ground-truth datasets for model evaluation.
- Design and run sampled audits, including evaluation-phase quality control and fair comparisons across methods or searched sources.
- Gate releases against model versions and maintain version inventories, evaluation records, and rollback triggers.
- Review model drift and overrides.
- Produce human-oversight and fairness or disparate-effect evidence.
- Measure workflow improvement using review time, throughput, backlog movement, and override and rework rates.
- Define the measurement events other teams must emit and confirm the review workspace records workflow events from first use.
- Maintain independence from model-building roles and do not approve thresholds or authorize releases based on your own evidence.
View Full Description & ApplyYou'll be redirected to the employer's site
