In this video, we will be reviewing how to use data and measures to evaluate success of an improvement effort. My name is Julia Kim and I'm a faculty member at the Johns Hopkins University School of Medicine in Pediatrics and the Armstrong Institute for Patient Safety and Quality. I serve as the Associate Vice Chair of Ambulatory Quality and Safety and the Co-Director of the Armstrong Institute Patient Safety and Quality Leadership Academy. I have worked on both outpatient and inpatient quality improvement projects. When discussing data and measures, it's always good to go back to definitions. What is quality improvement? The Health Resources and Services Administration defines quality improvement as systematic and continuous actions that lead to measurable improvement in health care services and measurable improvement in the health status of targeted patient groups. So, we can see that measurement is the foundation and the basis for quality improvement. There are several frameworks for quality improvement. One, is the Lean Sigma framework, where you define a problem, measure, analyze, improve and then control and sustain your improvement. There's also the Institute for Healthcare Improvement, with the three big questions addressing the aim, measures, and ideas for change and then the plan to study act cycles for continuous improvement. There's also the translating evidence into practice or TRIP model, where one summarizes the evidence, identifies local barriers and implementation, measure performance and ensures that all patients receive interventions. In each of these frameworks and other frameworks as well, you can see that measurement plays an important role, an important step in your approach to quality improvement. In defining your problem and in defining your aim for your quality improvement project, it is always good to consider the Institute of Medicine Aims for Improvement, as outlined in the 2001 Crossing the Quality Chasm report, that outlined the aims for Healthcare Improvement to be safe, effective, patient-centered, timely, efficient, and equitable. Similarly, it's also helpful to think about your aim and couch it as a smart aim and for it to be specific, measurable, achievable, realistic, and relevant, and timely. So when we get to the measurement aspect and we ask ourselves a question, how will we know that a change is an improvement? It boils down to measurement. Your key measure should be related to your aim. Also, measurement over time is an essential aspect to quality improvement and you want to ensure that your measure is feasible to collect and that's able to show improvement quickly. One tip is to select multiple measures and to choose different types of measures to accurately evaluate your changes. Three types of measures include outcomes measures, which are your clinical outcomes or the outcomes that reflect changes in the system. There's also process measures, which reflect the steps that need to be taken to achieve the outcome measures, and these are often identified as some of the key drivers to achieving your clinical or systems-based outcomes. There are also balancing measures or those measures that are not directly related to your quality improvement project, but will reflect whether or not there are unintended consequences of your interventions or ideas for change. And ideally, you'd use a balanced set of measures for all improvement efforts. When you think about selecting your measure, you want to be able to define your measure to be able to plot it on a graph. You want to be able to plot the measures on the graph to show the effect of the changes you are testing. You want to be able to select an appropriate metric and think about, is it best to define your measure as a ratio or a percentage, and ideally, you would be explicit with the numerator and denominator of your measure, or is it best to track your measure as a rate versus account? And if it's an event that you're trying to track that may not be as common, you can also consider tracking the time between events. Another tip would be to collect data as efficiently as possible. And if resources are not available to collect your measures on a daily basis for all patients, then you can use sampling strategies. For example, to collect data on 25 patients that present during a week or to collect data for a week out of the month. Also, you want to be able to plot the data often enough to be able to provide your teams with timely feedback using the graphs. This can often help to motivate the teams to continue with the improvement project. And in thinking about data collection and how to present the data, you can think about whether this is going to be monthly data, weekly data or daily data and often by collating the data monthly and presenting it in monthly graphs. This can minimize the variability of day to day results. Another tip would be to try to integrate the data collection into the daily workflow of the staff to minimize the burdens of collecting data for your quality improvement project. And if resources are available, think about incorporating automated data reports from an electronic health records system or to include the information technologists on your quality improvement team. One of the key aspects of measurement is to measure data over time, and there are a couple of ways that the data can be presented. And one of the most common and effective ways to demonstrate that data over time is through a run chart or a time series graph. And in this example of the run chart, we can see time along the horizontal X axis and we can see that your measurement, your metric, would be along the Y vertical axis, potentially as a percentage, as in this example and you would collect baseline data. And in this example, there are 10 data points and you want to have at least 10 data points in your baseline data. And then, you would look at the baseline median and extend that line going forward. And then, you would then track your interventions or ideas for change. And you can see an intervention one demonstrated, that it started right after Time point 10. And then, you would continue to plot your time points and then you would demonstrate when a second intervention potentially was implemented. And you would also note your goal line, and in this case, the goal was 70%. So, there's a lot of information that can be conveyed through this run chart. When interpreting the results of your run chart, you want to look for several different patterns in the data. And so, you can look for shifts in the data where there are at least six points above or below the center median line. You can try to look for astronomical data, or points that are so far off compared to the other data points, or you can look for trends in the data, where there are at least five consecutive points that either run upwards or downwards. So, these are some examples of signals for an effective change process taking place. So another way to demonstrate measurement of data over time is through the use of control charts. Similarly, we can see the key metric plotted over time. The difference with control charts are that they include upper control limits and lower control limits. And these control limits indicate three sigma from the mean. And they help us to differentiate between random common cause variation of a data versus special cause variations. So, there are also special cause variation tests that we can use to try to determine whether or not we are impacting and having effective change. And so, some key differences between the run chart and the control chart are that the run chart plots the median while the control chart plots the mean. The run chart does not include control limits while the control chart includes +/- 3 sigma upper control and lower control limits which can help us distinguish between common cause versus special cause variation. The run chart requires less data points, at least 10 baseline data points, control chart requires more. And we have signals in both run charts. We can identify shifts, trends, runs or astronomical points on the run chart or through the control charts, we can also identify shifts, trends, cycles or outliers. And here are some references that you can refer to find out some more information about how to use data and measurement in your quality improvement process.
