How Can Education Experiments Make Education Better?

In a previous post, I suggested that to make education better we should “Stop arguing about what people think will work in education and start experimenting to determine what actually works.”

There were a number of reader comments (thanks to Jeff, Stacie, and Suzanne in particular) that helped me to understand that the word “experiment” may carry some extra baggage that wasn't part of what I had in mind here.  In this post, I’ll try to clarify what I mean and offer an example.

How not to run education experiments

First, a few things I don’t mean when I talk about “education experiments”:

• I don’t mean we should bring scientists in white lab coats into the classroom so that they can better understand how our children’s brains work – that is cognitive science, not education science.
• I don’t mean we should bring academics in white lab coats into the classroom to test their speculative or radical new math curriculum for the very first time on our children for the benefit of future generations.

Don’t get me wrong – I think both basic and applied academic research are extremely important and have their place (with appropriate safeguards and informed consent all around). In particular, these kinds of research can benefit society greatly over the long term. But academic research is not what I had in mind here for two reasons:

1. I am more interested in research that has a good chance of benefiting the participating students in the short term, and
2. I am interested in research that leverages the teacher’s special knowledge and insight about the capabilities and needs of those particular students.

A different take on education experiments (with an example)

What I am proposing is more akin to action research, or design-based research.  To clarify what I have in mind, I’ll give an example from real life.

In my work as an educational designer and consultant, I have the pleasure of talking with some extremely dedicated teachers. Sometimes I meet them when they send a message, out of the blue, with a specific idea or question about how to do more for their students. Below is an example of one such message I received about a year ago.

Key points to note about the situation:

• This teacher has identified a problem: Something isn’t working right, and my students are struggling in math.
• This teacher has a hypothesis about the root cause of the problem: I believe some of my students lack number sense, which prevents them from understanding the more complex material in the math curriculum.
• This teacher has done some research (most likely on her own time) and identified an option that she believes could address the root cause better than the current curriculum alone: I have found an iPad math curriculum that I believe will solve the problem by developing my students’ number sense.
• This teacher has encountered a barrier that prevents her from trying the option: I don’t have all the technology I need (wifi) to use the app in my classroom.
• This teacher is actively seeking creative ways to overcome the barrier: Is there a way I can use the app-based curriculum without wifi? (In my experience, teachers are willing to go to great lengths to make something work if they believe it will help their students.)

This situation has all the trappings of a good old-fashioned experiment. What happens next?

Here are two common outcomes:

• Either: The teacher gets no support to overcome the final barrier, she can’t even try out the option she believes will help her students, and her students continue to struggle with math – probably for life (because math misunderstanding – like math understanding – is cumulative). Sadly, this is probably what happens to proactive teachers and their students most of the time. The teacher takes the ball to the 99 yard line and for any number of reasons can’t carry it the final yard alone so it sits there. The students miss out. The dedicated teacher gets a little more burnt out – maybe this is even the last straw and she leaves the profession. Everyone loses. 
• Or: The teacher finds a creative way around the barrier, tries out the math app with her students this year, and based on her results she decides whether to use it again (or try something else) with her class next year.  The teacher goes to great lengths to get her students more of what they need. Her twenty-five students benefit. She feels empowered - maybe she can make a difference through diligence, resourcefulness, and a lot of hard work. But…did her idea work? How well? What’s the evidence that it did or didn't? Did it work for some kids and not others? If so, which ones? What did she do to make it work? That teacher has a lot of valuable, actionable insight as a result of her experiment, but it was a private experiment and so the world will never know about it.

The answers to those last questions would have been handy six months later when I received this message from another teacher:

It’s déjà vu all over again.

Key points to note about this situation:

• This teacher has independently identified the same problem: Something isn’t working right, and my students are struggling in math.
• This teacher has independently produced the same hypothesis about the root cause of the problem: I believe some of my students lack number sense, which prevents them from understanding the more complex material in the math curriculum.
• This teacher has done some research (most likely on her own time) and independently identified the same option that she believes could address the root cause better than the current curriculum alone: I have found an iPad math curriculum that I believe will solve the problem by developing my students’ number sense.
• This teacher has encountered a slightly different barrier that prevents her from trying the option: I don’t have all the technology I need (iPads) to use the app in my classroom.
• This teacher is also actively seeking creative ways to overcome the barrier: Is there a way I can adapt the app to a whole-class format so I can make it work with the technology I do have available?

The teacher from California observed a pattern, had a hypothesis, came up with a test of that hypothesis, and – if she was persistent and lucky enough – was able to run the experiment.  But no one except perhaps a few colleagues will ever hear about it, whether it was successful or not. As a result, the teacher in Tennessee has to do all of the same work over again, from scratch, without guidance or data to help her make informed decisions. Which raises a few questions:

• If these two teachers face the exact same problem with teaching math, how many others are there? Tens of thousands? Hundreds of thousands?
• How much teacher time, energy, and good will are we wasting with all of this duplicated effort?
• How many more teachers would use an effective option that had already been vetted by another teacher if they knew about it and didn’t have to re-invent it from scratch or take the risk of "being first"?
• Looking beyond first grade math, how many students would benefit if every teacher didn’t have to reinvent the wheel like this on every big and little problem they encounter in the classroom? Presumably every single student. That includes your children, and mine.

Here’s the punch line:
This is going on all the time.  Millions of such private experiments are presumably being conducted by teachers every day in classrooms across the country and around the globe. The only thing that’s missing to capture the value of that activity is a bit of systematic record keeping and a way to share results.

Where do we go from here?

I hope that helps clarify what I meant when I said we need to stop arguing about what people think will work in education and start experimenting to determine what actually works. 

In fact, I now realize that I mis-spoke when I first said that. We don't need to start experimenting - that part is already happening. As a first step we simply need to start sharing what teachers have discovered about what works and doesn't work as they try – sometimes desperately and at significant personal cost – to give our children more of what they need to succeed in school and in life.