Pragmatic Evaluation of Large Language Models in Healthcare

Overview

Large Language Models (LLMs) and LLM-based tools are becoming ubiquitous in healthcare. However, there is a dearth of analyses evaluating the efficacy of these tools. This tutorial has three learning objectives for participants:

  1. gain a rigorous conceptual understanding of how LLMs are built;
  2. become familiar with the various frameworks available in the literature for LLM evaluation
  3. observe three real-world case studies that demonstrate the use of these frameworks (and other open-source tools) to evaluate LLM deployments of increasing complexity.

Program

  1. Introduction to LLMs in healthcare
  2. Architecture of an LLM-based application
  3. Review of language model evaluation frameworks, benchmarks, and metrics
  4. Case studies
    • Translation evaluation
    • Patient-facing web chatbot
    • Hospital knowledgebase search assistant

Slides

Slides will be made available at the conclusion of the tutorial.