Director, Clinical AI Research & Evaluation
AstraZeneca · United States, Cambridge Kendall SQ, Massachusetts
Job Title: Director, Clinical AI Research & Evaluation
Introduction to role:
About AISI AI Science & Innovation (AISI) sits at the centre of AstraZeneca’s R&D AI transformation within Enterprise AI Unit (EAI). Our remit is to build, buy, and deliver the AI models and agents that change pipeline outcomes across discovery, translational science, biomarkers, and clinical development, ultimately improve patients’ lives.
The opportunity
We are building an end-to-end Enterprise AI engine that unites data foundations, AI models, platforms, and business-facing applications to accelerate results across the value chain. Success comes from reusing what already works, sharing ideas across teams, and scaling impact rather than building in isolation.
Role overview
AstraZeneca is building an AI capability for Clinical Development that will change how trials are designed, conducted, monitored, and analyzed. We are hiring a Director, Clinical AI Research & Evaluation to lead the scientific foundation that makes clinical AI systems trustworthy, measurable, and ready to scale into regulated practice. The role sits in the Clinical AI Applied Science & Translation team, based in Boston, MA. You will set the research agenda for how clinical AI systems are evaluated, how human professionals interact with AI in clinical workflows, and what evidence standards must exist before a system can be trusted at scale. This is a research leadership role with delivery accountability. You will define the methodology as we practice it, while remaining close enough to the work to credibly challenge it. You will direct a team of scientists and engineers, partner with product owners and Engineering, and establish evaluation and human-factors practices that extend across the Clinical Operations and Development AI portfolio. The initial focus for this role is an agentic system being built to reduce the administrative burden for research associates conducting trials at sites but with the expectation that patterns discovered and built here will scale to additional clinical AI system we develop along the way. The strongest candidates combine deep research expertise in AI evaluation or human computer interaction with the practical grounding to close the distance between principled methodology and working with and building regulated software.
Accountabilities:
Research Leadership
- Define and own the scientific methodology for evaluating clinical AI systems: evaluation harnesses, benchmark construction, failure mode taxonomy, red-team protocols, and monitoring strategies across the system lifecycle. Build a structured catalog of ways the system can fail clinically and run structured testing before any deployment.
- Advance methods for evaluation in clinically complex conditions such as noisy or delayed ground truth, expert disagreement, multi-step agentic workflows, human in-the-loop performance, and automation bias or where simple metrics fail.
- Develop human factors and workflow-integration methods beyond usability testing: how AI output shapes human judgment, what oversight mechanisms are effective, how clinical professionals calibrate trust in AI-assisted work, and what human review effectively ensure patient safety.
- Publish and communicate findings externally where appropriate to build AstraZeneca’s scientific credibility and attract research talent.
Applied Delivery
- Design and implement evaluation frameworks that determine when a system is ready to move from prototype to controlled pilot, from pilot to deployment, and from non-GxP to regulated use.
- Participating in defining human review requirements for user facing AI outputs: what requires sign-off before action, what can be surfaced directly, and how oversight should be structured in the validation plan.
- Build evaluation assets such as test sets, simulation environments, scoring rubrics, regression suites, and analysis pipelines, some or all of which are reusable across
- Lead hands-on failure analysis: fabrications/falsifications, unsupported claims, data leakage, subgroup performance, and unsafe workflow behavior.
- Be part of a team of scientists and engineers focused on evaluation, human factors, and responsible deployment.
Cross-Functional Governance
- Work with Quality, Regulatory, Privacy, Security, and R&D IT to ensure evaluation of evidence meets GxP, data integrity, and inspection-readiness requirements from the outset.
- Advise product owners and leadership on evidence-based go or no-go decisions; communicate residual risk clearly.
- Create reusable evaluation standards and clinical AI governance practices that can be adopted across AISI’s Clinical Development portfolio and contribute to broader industry standards.
- Represent the team’s scientific position in external forums, standards bodies, and regulatory engagements where relevant.
Essential Skills/Experience:
- PhD in Machine Learning, Computer Science, Human-Computer Interaction, Statistics, Biomedical Informatics, or a related quantitative discipline; or an MD or equivalent experience with a research record that demonstrates comparable rigor.
- 8 or more years of post-PhD (or equivalent) experience in AI/ML research, evaluation science, or human factors in healthcare, life sciences, clinical research, or another safety-critical domain.
- Deep expertise in evaluation methodology for AI/ML systems: experimental design, statistical inference, uncertainty quantification, measurement reliability, and evaluation of LLM-based or multi-step agentic workflows.
- Demonstrated research leadership: ability to set a scientific agenda, direct a team, produce research outputs, and translate principled methodology into practical, shipping systems.
- Current, hands-on coding ability in Python and SQL, including evaluation pipelines, data analysis, statistical testing, and version-controlled collaborative development.
- Strong grounding in human-computer interaction or human factors and its application to AI-assisted decision-making in high-stakes environments.
- Excellent communication skills across technical, clinical, quality, regulatory, and executive audiences; ability to represent uncertainty and residual risk without overstating or obscuring.
- Demonstrated ability to lead cross-functional work through influence, including go or no-go recommendations under ambiguity.
- Experience with LLM evaluation platforms, LLMOps/MLOps & observability for deployed AI systems.
Desirable Skills/Experience:
- Direct experience with clinical trial conduct, site monitoring, risk-based monitoring, medical monitoring, or clinical data management in a pharmaceutical, biotech, or CRO environment.
- Experience producing audit-ready evidence under GxP requirements, including model risk management, audit trails, traceability, and inspection readiness for regulated AI systems.
- Experience designing or conducting prospective validations, implementation studies and/or human-computer interaction studies in live clinical or healthcare workflows.
- Peer-reviewed publications, standards body contributions, open source benchmarks, or regulatory and industry consortium engagement in AI evaluation, human factors for AI, or clinical AI governance.
- Familiarity with causal inference, calibration, subgroup analysis, weak supervision, active learning, or evaluation methods for learning with imperfect labels.
- Experience working with Quality or regulatory functions on computerized system validation, validation plans, or submission-relevant AI evidence.
What success looks like
- The team establishes an evaluation standard for clinical AI that is recognized internally as the bar for production readiness and externally as a principled contribution to the field.
- Clinical experts, Quality partners, and regulatory functions trust that evaluation evidence reflects real clinical risk rather than optimistic benchmarks.
- Every agentic release by the team is tra