Dashboarding for Machine Learning-based Clinical Decision Support Implementation and Monitoring Toolkit

• Datasets and Research Tools
• HIPxChange
• Health IT
• Learning Health System
• Older Adults
• Quality and Safety

Background

Machine learning-based clinical decision support (ML-CDS) tools hold enormous promise to simultaneously improve clinical care and reduce clinician burden. ML-CDSs can distill the abundance of data in modern electronic health records (EHRs) into an actionable recommendation to providers to inform medical decision making, and they can do so without the provider needing to perform dozens of time-consuming screenings for myriad patient conditions. In this way, ML-CDSs can streamline the process of identifying the right care for the right patient and presenting that information to providers at the right time. However, many challenges with ML-CDS still exist, and these tools require monitoring during and after implementation to ensure that they are functioning correctly. Threats to correct functioning include both endogenous and exogenous factors such as seasonal changes in health conditions, regulatory demands, and workforce pressures, all of which can cause upstream and downstream changes to the work system that alters the efficacy of ML-CDS interventions. Recently, the need for robust monitoring of ML-CDS interventions has been a topic of discussion (Bedoya et al, 2022).

This toolkit provides one possible response to this need. It contains a rough template for a dashboard to monitor ML-CDS tools during and after implementation. This dashboard was developed in the context of a ML-CDS intervention in the Emergency Department (ED) to identify patients at high risk for an outpatient fall and prompt providers to refer qualifying patients to a Mobility and Falls Clinic in the health system to ameliorate the patients’ fall risk. The dashboard we developed takes a “zoom-in/zoom-out” approach that allows us to zoom out and track both overall patient flow through the referral process as well as zoom in at specific decision-making points. We use an alluvial plot to get a bird’s eye view of how many patients are completing the referral visit as well as when and why patients are not completing the referral. At the individual decision points, we use statistical process control charts to distinguish between statistical noise and signal to identify data drift or other endogenous threats to our process.

Data Model and Code Flow

The data needed for similar dashboards will vary by the ML-CDS being monitored, but generally, data should be at the patient encounter level of cardinality and follow the CDS recommendation from the time the patient was identified through to either clinical completion (e.g. a completed referral) or abandonment (e.g. a patient being determined to be contraindicated for the recommended clinical therapy after further clinician review). In this sample dashboard, we used data from the EHR, supplemented with manual review data from REDCap. Data were extracted from the EHR, a combination of free-text and discrete data were parsed using regular expressions to guess the most likely patient outcome (e.g. screened out due to being in hospice care), but that guess was then imported into REDCap where implementation team members reviewed the patient’s chart to verify or correct the patient outcome. EHR and REDCap data were merged again as part of the dashboard code.

Code for the dashboard is modularized into four primary files. First, the sqlgen file contains R functions to compile dynamic SQL to pull the necessary data from the EHR. These functions are not used in this example but are used in our production dashboard. The data_io file contains R functions for retrieving and cleaning data. For the sample dashboard, data are contained in csv files, but we retained as much of the code for data parsing as possible to demonstrate the blending of EHR and REDCap data. The graphs file contains functions for several of the graphs used throughout the dashboard. Finally, the RMD, or RMarkdown file contains the code to generate an html dashboard using the R knitr package. For multi-site implementations, it can be useful to script out multiple sites at once using a file similar to the Script Dashboards file. In our production version, this file also handles uploading new data to REDCap.

Who should use this toolkit?

This toolkit is intended for clinical informaticists and clinical decision support (CDS) managers to monitor new CDS tools, especially those based on Machine Learning algorithms.

What does the toolkit contain?

This toolkit contains:

1. R code using RMarkdown to make a reproduceable dashboard
2. CSV files of fake data to demonstrate the necessary data input for the dashboard
3. Data Dictionary

How should these tools be used?

The materials in this toolkit can be used to:

• Start your own ML-CDS monitoring dashboard for specific ML-CDS tools
• Incorporate alluvial plots or control charts into existing R-based dashboards

Customizing this toolkit

This toolkit is only a template, and you will need to use data scientist resources within your organization to customize this code to identify and quantify outcomes germane to your specific ML-CDS interventions. The code is modularized in such a way to make it relatively to adapt the content for use in your organization.

Development of this toolkit

The Dashboarding for Machine Learning-based Clinical Decision Support Implementation and Monitoring was developed by researchers and clinicians (Principal Investigator: Brian W Patterson, MD, MPH) from the Berbee-Walsh Department of Emergency Medicine at the University of Wisconsin-Madison School of Medicine and Public Health.

This project was supported by the Agency for Healthcare Research and Quality grant number R18HS027735 (Patterson, PI). Additional support was provided by the University of Wisconsin School of Medicine and Public Health’s Health Innovation Program (HIP). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality or other funders.

References

• R Core Team. R: A Language and Environment for Statistical Computing. 2020. https://www.r-project.org/
• Wickham H. ggplot2: Elegant Graphics for Data Analysis. 2016. https://ggplot2.tidyverse.org/
• Allaire JJ, Xie Y, McPherson J, et al. rmarkdown: Dynamic Documents for R. 2021. https://github.com/rstudio/rmarkdown
• Xie Y, Dervieux C, Riederer E. R Markdown Cookbook. 2020. https://bookdown.org/yihui/rmarkdown-cookbook
• Wickham H, Averick M, Bryan J, Chang W, D’Agostino McGowan L, Francois R, Grolemund G, Hayes A, Henry L, Hester J, Kuhn M, Lin Pedersen T, Miller E, Milton Bache S, Müller K, Ooms J, Robinson D, Paige Seidel D, Spinu V, Takahashi K, Vaughan D, Wilke C, Woo K, Yutani H. Welcome to the Tidyverse. J Open Source Softw. 2019 Nov;4(43):1686.
• Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009 Apr;42(2):377-81.
• Bedoya AD, Economou-Zavlanos NJ, Goldstein BA, Young A, Jelovsek JE, O’Brien C, Parrish AB, Elengold S, Lytle K, Balu S, Huang E, Poon EG, Pencina MJ. A framework for the oversight and local deployment of safe and high-quality prediction models. J Am Med Inform Assoc. 2022 Aug 16;29(9):1631-1636.
• Jacobsohn GC, Leaf M, Liao F, Maru AP, Engstrom CJ, Salwei ME, Pankratz GT, Eastman A, Carayon P, Wiegmann DA, Galang JS, Smith MA, Shah MN, Patterson BW. Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments. Healthc (Amst). 2022 Mar;10(1):100598.
• Berwick DM. Controlling variation in health care: a consultation from Walter Shewhart. Med Care. 1991 Dec;29(12):1212-25. 

Toolkit Citation

Hekman DJ, Maru AP, Patterson BW. Dashboarding for Machine Learning-based Clinical Decision Support Implementation and Monitoring. Berbee-Walsh Department of Emergency Medicine at the University of Wisconsin-Madison School of Medicine and Public Health. Madison, WI; 2023. Available at: http://www.hipxchange.org/CDSDashboard