You might be undertaking a project to improve an aspect of the service you provide and are being challenged to demonstrate the impact. How can you do this? Or maybe your current performance report is more confusing than enlightening and you spend a lot of time in meetings arguing the toss. How can you really know whether things are getting worse or better?
Both those scenarios revolve around the issue of measuring something. And even if we are happy to tackle this, and let’s be honest most of us would rather not, we run into a number of problems. Any one of these problems can prevent us from being able to answer those questions. And this is why a few of us at the NHS Institute for Innovation and Improvement created a short video to point out these problems and suggest how they can be avoided. You can view the video here 1.
The first problem we face is that everyone has a different view as to what we are measuring and why. Leif Solberg 2 and colleagues wrote a super paper back in 1997 that addressed this. They said that there were 3 basic reasons why we measure: for accountability; for research and for improvement. And each of those has a distinctive approach which we need to heed. We are thinking in this article about knowing whether we are getter better or not, this is measurement for improvement. Those who come from one of the other perspectives can struggle to make the shift. For example those who inhabit the world of performance and targets (accountability in Solberg’s language) sometimes don’t pay enough attention to the way data is collected. Those who come from a research background can struggle with the absence of their favourite statistical tests. Neither group can be comfortable with the idea of plotting data over time.
So why are we wanting to measure? What is it that we want to know exactly? Stacey Barr who writes on performance measurement has a great blog post 3 on just this. We need to be very specific about what we want to achieve, using plain simple English and not resort to ‘jargon’ phrases such as ‘best quality’. One of the main reasons why project outcomes are so hard to measure is that they are not expressed in a way that makes them amenable to measurement. Follow Stacey’s advice and this problem is solvable. She also has lots to say on the other problems we’ll cover too so have a browse on her website.
Once we’ve sorted that out we can move onto the next problem which is choosing the right things to measure. If we’ve followed Stacey’s advice above we may already have that one covered. If we have phrased our aim as an outcome, the most relevant measure will be an outcome measure. But as Donabedian reminds us in his Quality Model 4 to get a good outcome you need to consider both the inputs and also the processes by which care is delivered. So how are we doing on those? Tracking both inputs and process compliance could be useful things to do. Finally, we work in a health and social care system so we always have to be mindful that any changes we make could have a knock-on effect elsewhere. Identifying and tracking potential adverse effects, a ‘balancing’ measure, is the final type of measure we should consider.
At this point some would think that we’ve reached the end of our problems. After all, we know what we want to measure now so let’s just ask for the data and away we go. But both defining and collecting data can contain traps for the unwary. Neil Pettinger discusses some of the pitfalls about definitions in his blog 5 on measuring bed occupancy and Davis Balestracci reminds us in another post 6 that measurement itself is a process and we need to be careful how we go about it.
So take some time to ensure you have common agreement on definitions. And test the data collection process you are using. Don’t assume any process will work perfectly first time. The first data we should collect will give us an idea of where we are now. This is our baseline and we judge the impact of the changes we have made against this position. This means of course that we have to start collecting our data before we make changes. It sounds so obvious that you are probably thinking why have I mentioned it. Sadly many projects suffer because enthusiastic staff have rushed to make a change without knowing how they are doing now. Of course it is not always possible to obtain baseline data, for example if you are starting a new service, but you should make a determined effort to do so.
When we do display data about our performance, it will show variation. How do we handle this? Most of the time we try to remove it. After all that’s what using totals or averages do for us, they ‘clean up’ our data and get rid of all that messy variation. However this can be dangerous. Many years ago Dr Walter Shewhart (creator of the SPC chart) coined 2 rules for presentation. They are:
Rule One: Data should always be presented in a way that preserves the evidence in the data
Rule Two: When an average, standard deviation or histogram is used to summarize data, the user should not be misled into taking action they would not take if the data were presented in a time series
If I were to paraphrase Shewhart’s rules I’d say “Don’t use 2 point comparisons or averages on their own”. Shewhart was concerned to ensure that the ‘messy’ variation we want to airbrush out stayed in full view. Why? Because it tells us a lot about how the process we are measuring typically performs. And we can use that knowledge to help us make it a better process.
So how should we present time series data? This is where run charts and Statistical Process Control (SPC) charts come in. Run charts are simple to use and anyone can create them. Just follow the guidance in this article 7 to get started. SPC charts appear more complex (they have 2 extra lines) but are no more difficult to use once you understand how they are constructed. The best reference on these is a book called Understanding Variation 8 by Don Wheeler.
So now we have addressed our problems, the final step is to make a decision based on the information in front of us. And, if we really have chosen our measures well and obtained good data, that’s simply not a problem! The rules or tests associated with run or SPC charts will guide us to whether we have just random variation in our data or whether there are specific (or ‘special’) causes lurking in there as well. That knowledge drives the type of decision we make. We can find and address special causes with a specific contingency plan. For unacceptable random (‘common cause’) variation we need to think about revising or redesigning the whole process.
If you’ve been counting up the problems or steps mentioned in this article so far, you will have got to six. And there are seven steps to measurement in the video referenced earlier. So what’s the seventh? It’s simply this. Keep going. Don’t think that because you’ve studied your data once and made some decisions, you can now forget all about it. Continue to collect, plot and review your data on a regular basis. Once you get into the swing, you’ll find that won’t be a problem either.
Mike Davidge, Director, NHS Elect
7 “The run chart: a simple analytical tool for learning from variation in healthcare processes”; Perla R, Provost L, Murray S; BMJ Qual Saf 2011;20:46e51. doi:10.1136/bmjqs.2009.037895
8 “Understanding variation” by Don Wheeler obtainable via http://www.spcpress.com