Comparative Effectiveness

(NB: this post eventually is about education, if you stick with it.)

Interesting article from the New York Review of Books on the sometimes controversial notion of “comparative effectiveness research” in healthcare.  In short, the notion here is that doctors, other  providers, and insurers should use data on the relative effectiveness of different interventions when making decisions about both the delivery and financing of care.  While I am definitely not a healthcare expert, I have been following the reform debates fairly closely, and most of the folks I read and agree with (Ezra Klein, Jon Chait for example) assert that making comparative effectiveness research more sophisticated and widespread is critical to improving outcomes and lowering costs.  The easiest example of this: prescribing generic drugs that are functionally identical to – yet a fraction of the cost of – their branded analogues.  Same outcomes, lower cost.

Most of the other examples, unfortunately, are less straightforward.  Whereas the concept of using data on effectiveness to drive patient care sounds like a no-brainer in theory, it’s (shocking!) hard to implement.  The complexity is the issue dealt with in the linked article.  One of the biggest challenges is the standardization of complex procedures (sorry for the extreme detail here):

“… Medicare specified that [a procedure] was a “best practice” to tightly control blood sugar levels in critically ill patients in intensive care. That measure of quality was not only shown to be wrong but resulted in a higher likelihood of death when compared to measures allowing a more flexible treatment and higher blood sugar. Similarly, government officials directed that normal blood sugar levels should be maintained in ambulatory diabetics with cardiovascular disease. Studies in Canada and the United States showed that this “best practice” was misconceived. There were more deaths when doctors obeyed this rule than when patients received what the government had designated as subpar treatment …”

Keep the healthcare discussion in mind as debates about measuring the effectiveness of teachers heats up.  A few key patterns to look for in the healthcare debates:

1) Method of Measurement – how is relative effectiveness defined, and how is (are) that (those) metric(s) measured?

2) Inputs vs. Outputs – what is the balance of measuring inputs and outputs?  If outputs are weighted more heavily than inputs, how is that weighting determined?

3) Growth – how is growth measured?  Is it based on individuals or on the aggregate?

A lot of folks argue that teacher effectiveness measures could become unfair to teachers who have larger proportions of students that either are multiple grade levels behind in literacy/numeracy or have learning disabilities.  The standard response is that outputs will take into account growth, not snapshot data, thus rendering the argument moot.

I would say that getting a 9th grader to read at grade level – if s/he currently reads at a 4th grade reading level – is a “complex procedure.”  We definitely should measure the effectiveness of a teacher who is tasked with this challenge, but we should learn what we can from the codification of complex procedures in other fields.  I like healthcare as an analogue, because like in education it involves fallible and complex humans on both sides of the delivery system (i.e. patient-doctor vs. student-teacher).  And like in healthcare, some education challenges probably require specialists.

All of this to say … measuring effectiveness: important!  Being particular about what is measured and how the measurement is being done: equally important!

2 Responses to Comparative Effectiveness

  1. Pingback: Comparative Effectiveness, cont. « Meeting the Turnaround Challenge

  2. The health field is lucky, even though reformers may not see it that way, because the self-regard of doctors means that their professional autonomy must be protected. Agreeing with your conclusion, I also have to ask:

    1. how many examples to we have of educators bringing freshmen who read a 4th grade levels to grade levels?
    2. How many of the successes were accomplished be generalists in schools that are funded at the level of generalists?
    3. Wouldn’t it be more cost effective to create a class of specialists for those tough cases?
    4. How many medical specialists treat 140 cases a day?
    5. Would we tell victims of PTSD or rape victims and their providers to just raise expectations and measure their results and they will get well? Why do we dump that disrespect on students and their teachers?

    Health care has tried data-DRIVEN accountability and my understanding is that it widely and dramatically failed. I don’t know enough to make judgements on the Groopman-Gwande debate, but let’s not forget that health reform today is being pushed toward data–INFORMED acccountablity and evidence-based decision-making. Data-DRIVEN health reform is just as much a nonstarter and data-DRIVEN education reform, and my understanding is that the distinction holds across the history of diverse professions. My understanding is that DATA-DRIVEN contradicts basic human and social dynamics, as proven by a large body of social science and cognitive science.

    Measurment is important. But people must control the metrics, not be controlled by them.

    And thanks for the reminder that teaching reading IS rocket science.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: