Data Use, Technology, and Educational Leadership

When intuitions about assessment are wrong

Let’s say you’re a teacher or school administrator who knows how to incorporate data well into educational decision making. What if everything you “know” about the data is wrong?

I just finished re-reading a thought-provoking article in PDK by Henry Braun and Robert Mislevy. Henry is a faculty member here at BC and had recommended it me during a conversation about “stuff administrators should know about data.” According to the authors (and they’re probably right) most of what the average person knows about tests and their properties is “intuitive.” In other words, it’s not based on scientific thought or careful study. More likely, it’s based on what we remember from the design of Friday quizzes when we were middle school students.

I have two favorite anecdotes from them. The first is that while an “intuitive” knowledge of physics is okay for playing catch or building a, it’s not good for building a bridge or shooting a rocket to the moon. The other is that while intuitive understandings about testing are okay for the quiz in Seventeen magazine, it’s not great for things like high-stakes testing policies or measuring change in achievement among populations.

In fact, the authors present an insightful list of ways in which the conventional wisdom about data in schools may be misunderstood or only half true. Are our two math tests actually comparable? Not always. Are multiple choice questions only about factual recall? No, they can tell you much more than that, if the right work is put into them. How can you tell if an item is good or bad? Sometimes by looking, but sometimes not.

Braun and Mislevy do address problems in large scale (policy) decisions involving data and testing, but I think they also raise practical questions in schools. Many working in schools today face a knowing doing gap. “We know that the data we have are not appropriate for all decisions, but what else are we going to do?” The responses to this problem aren’t easy, and for the practicing teacher or administrator it means looking around and evaluating one’s workplace.

Do the people around you know that not all data should “drive” decisions? How open are they to examining multiple forms of data? To aligning data to a shared idea about teaching and learning? To seeking data sources (including teacher experience) that go beyond the big state test? The list could go on; feel free to add to it via comments.


One comment on “When intuitions about assessment are wrong

  1. Pingback: My weekend reading: Assessment as a Policy Tool | Datapulted

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


This entry was posted on June 22, 2012 by and tagged .