With A Little Help From My Friends

Accompanying blog post for the Posit Community talk: “With A Little Help From My Friends: Tools and insights for developing and deploying in the hospital”.

This is a summary of “With A Little Help From My Friends: Tools and insights for developing and deploying in the hospital”, a talk given on November 15, 2022 as part of the Posit Enterprise Community Meetup series. A recording of the talk is available here and a PDF of the accompanying slides can be found here.

A success story

Here at DSAA, we developed CHARTwatch, an early warning system for detecting patient deterioration. The system runs every hour:

• First, CHARTwatch ingests laboratory values, vital measurements, and demographics.

• Then, CHARTwatch classifies each visit into the following group: High risk vs Medium risk vs Low risk.

• The risk group predictions are then delivered through different methods:

  • Emails

  • Updates to a front-end tool

  • Alerts sent to phones

Finally, on August 2020, CHARTwatch went live and we were able to deploy to the General Internal Medicine (GIM) ward!! Developing and deploying CHARTwatch was a difficult feat and wouldn’t have been possible without the help of a few friends…

Tools for development

Database connections

There are various data systems in the hospital, all with their own quirks and intricacies. In order to make it easier for the Advanced Analytics team to work with the different hospital databases, we developed chartdb, an internal R package.

• The connection functions follow the same pattern, making it easier for data scientists and data analysts to work with databases.

con_a <- chartdb::connect_databaseA(username = ..., password = ...)
con_b <- chartdb::connect_databaseB(username = ..., password = ...)
con_edw <- chartdb::connect_edw(username = ..., password = ...)
con_soarian <- chartdb::connect_soarian(username = ..., password = ...)
con_mak <- chartdb::connect_mak(username = ..., password = ...)
con_syngo <- chartdb::connect_syngo(username = ..., password = ...)

A reproducible environment

A one-sentence horror story: “It works on my machine.”

• To improve reproducibility, we use renv, an R package for R dependency management.

• When setting up your renv environment, the renv.lock will keep track of the different packages (and their versions!!). If a new person needs to work on the project, they can use the renv.lock file to download the same packages and the same package versions! Woohoo!

Package-based development

Why write a package?

• Plenty! Of! Reasons!

• In particular, writing a package makes it easy to share code and knowledge with others.

• And, writing a package greatly reduces the amount of times you’re copy-pasting code. Really, it’s a win-win-win situation!

Environment! Environment! Environments

We typically work with 3 different environment.

• The development environment is the one on your local computer or your development server.

• The staging environment is as close to the “real” deployment environment as possible.

• The production environment is where things actually get deployed. Speaking of production environments, now’s a great time to talk about the tools we need for deployment. Before doing that though, time for…

An interlude

A multi-sentence horror story:

In earlier attempts to deploy CHARTwatch, we were using multiple CRON jobs… that called different bash scripts… which then called different Python/R/Java scripts. Oh, and we had no separate environments. Or rather, we only had one environment, where test environment = staging environment = production environment

Tools for deployment

Interlude over! Let’s looks the tools we need for deployment.

Authentication

We rely on Posit Connect for this. Posit Connect works with the hospital Active Directory, which enables administrators to manage permissions and access. What does this mean?

• Developers don’t need to keep track of an extra server username and password.

• End-users can log in with their hospital credentials!

Scheduling

Posit Connect also gives us the ability to schedule scripts through their admin interface.

Downtime

Knowing when things don’t work is crucial for deployment.

• We use jarvis, our own internally-developed R package, to email and Slack notifications alerting us of issues.

• Each project will also have downtime protocol which defines the steps that various team members must take when something isn’t working.

  • For example, when a CHARTwatch downtime affects end-users, an email alert is sent to the entire hospital. Not all projects will require something like this, but since CHARTwatch is embedded in clinical care, many need to be aware of the downtime.

A secure way to download internally-developed packages

We need to limit who can access the hospital network. Posit Package Manager is a repository package management that lets us download packages while being disconnected from the Internet.

An implementation plan

Deploying an early warning system in the hospital involves working with many different groups. All worked together to develop the implementation plan. Some takeaways from the implementation plan:

• Consider existing resources! There’s no point in re-implementing a workflow that already exists.

  • For example, notifications were designed so that they fit within existing processes. Emails to charge nurses were sent at times that made the most sense based on their shifts.

• Have a silent deployment period! This is useful for identifying unexpected bugs.

  • One bug we found in silent deployment was caused by a particular NAsty lab…

Find out more

This is a very very very very brief overview of CHARTwatch and how we develop and deploy models in the hospital. Check out the reading list below for more details.

• To learn more about CHARTwatch’s model development and validation, see: “Preparing a Clinical Support Model for Silent Mode in General Internal Medicine” by Nestor et al. (2020).

• To learn more about the process changes that were required for CHARTwatch’s implementation, see: “Implementing Machine Learning in Medicine” by Verma et al. (2021)

• To read all about the technical infrastructure that lead to CHARTwatch’s deployment, see: “From Compute to Care: Lessons Learned from Deploying an Early Warning System into Clinical Practice” by Pou-Prom et al. (2022)

• To find out all about the data that is used to train CHARTwatch, see: “GIM, a dataset for predicting patient deterioration in the General Internal Medicine ward (version 1.0.0)” by Kuzulugil et al. (2022)

• This white paper by Signal1: “The Burden of Clinical Deterioration and How One Hospital is Tackling it with Machine Learning”