Monday, July 29, 2013

How Can Education Experiments Make Education Better?

The Scientists In a previous post, I suggested that to make education better we should “Stop arguing about what people think will work in education and start experimenting to determine what actually works.”

There were a number of reader comments (thanks to Jeff, Stacie, and Suzanne in particular) that helped me to understand that the word “experiment” may carry some extra baggage that wasn't part of what I had in mind here.  In this post, I’ll try to clarify what I mean and offer an example.


How not to run education experiments


First, a few things I don’t mean when I talk about “education experiments”:
  • I don’t mean we should bring scientists in white lab coats into the classroom so that they can better understand how our children’s brains work – that is cognitive science, not education science.
  • I don’t mean we should bring academics in white lab coats into the classroom to test their speculative or radical new math curriculum for the very first time on our children for the benefit of future generations.
Don’t get me wrong – I think both basic and applied academic research are extremely important and have their place (with appropriate safeguards and informed consent all around). In particular, these kinds of research can benefit society greatly over the long term. But academic research is not what I had in mind here for two reasons:
  1. I am more interested in research that has a good chance of benefiting the participating students in the short term, and
  2. I am interested in research that leverages the teacher’s special knowledge and insight about the capabilities and needs of those particular students.

A different take on education experiments (with an example)


What I am proposing is more akin to action research, or design-based research.  To clarify what I have in mind, I’ll give an example from real life.

In my work as an educational designer and consultant, I have the pleasure of talking with some extremely dedicated teachers. Sometimes I meet them when they send a message, out of the blue, with a specific idea or question about how to do more for their students. Below is an example of one such message I received about a year ago.



Key points to note about the situation:
  • This teacher has identified a problem: Something isn’t working right, and my students are struggling in math.
  • This teacher has a hypothesis about the root cause of the problem: I believe some of my students lack number sense, which prevents them from understanding the more complex material in the math curriculum.
  • This teacher has done some research (most likely on her own time) and identified an option that she believes could address the root cause better than the current curriculum alone: I have found an iPad math curriculum that I believe will solve the problem by developing my students’ number sense.
  • This teacher has encountered a barrier that prevents her from trying the option: I don’t have all the technology I need (wifi) to use the app in my classroom.
  • This teacher is actively seeking creative ways to overcome the barrier: Is there a way I can use the app-based curriculum without wifi? (In my experience, teachers are willing to go to great lengths to make something work if they believe it will help their students.)
This situation has all the trappings of a good old-fashioned experiment. What happens next?

Here are two common outcomes:
  • Either: The teacher gets no support to overcome the final barrier, she can’t even try out the option she believes will help her students, and her students continue to struggle with math – probably for life (because math misunderstanding – like math understanding – is cumulative). Sadly, this is probably what happens to proactive teachers and their students most of the time. The teacher takes the ball to the 99 yard line and for any number of reasons can’t carry it the final yard alone so it sits there. The students miss out. The dedicated teacher gets a little more burnt out – maybe this is even the last straw and she leaves the profession. Everyone loses. 
  • Or: The teacher finds a creative way around the barrier, tries out the math app with her students this year, and based on her results she decides whether to use it again (or try something else) with her class next year.  The teacher goes to great lengths to get her students more of what they need. Her twenty-five students benefit. She feels empowered - maybe she can make a difference through diligence, resourcefulness, and a lot of hard work. But…did her idea work? How well? What’s the evidence that it did or didn't? Did it work for some kids and not others? If so, which ones? What did she do to make it work? That teacher has a lot of valuable, actionable insight as a result of her experiment, but it was a private experiment and so the world will never know about it.
The answers to those last questions would have been handy six months later when I received this message from another teacher:
It’s déjà vu all over again.

Key points to note about this situation:
  • This teacher has independently identified the same problem: Something isn’t working right, and my students are struggling in math.
  • This teacher has independently produced the same hypothesis about the root cause of the problem: I believe some of my students lack number sense, which prevents them from understanding the more complex material in the math curriculum.
  • This teacher has done some research (most likely on her own time) and independently identified the same option that she believes could address the root cause better than the current curriculum alone: I have found an iPad math curriculum that I believe will solve the problem by developing my students’ number sense.
  • This teacher has encountered a slightly different barrier that prevents her from trying the option: I don’t have all the technology I need (iPads) to use the app in my classroom.
  • This teacher is also actively seeking creative ways to overcome the barrier: Is there a way I can adapt the app to a whole-class format so I can make it work with the technology I do have available?
The teacher from California observed a pattern, had a hypothesis, came up with a test of that hypothesis, and – if she was persistent and lucky enough – was able to run the experiment.  But no one except perhaps a few colleagues will ever hear about it, whether it was successful or not. As a result, the teacher in Tennessee has to do all of the same work over again, from scratch, without guidance or data to help her make informed decisions. Which raises a few questions:
  • If these two teachers face the exact same problem with teaching math, how many others are there? Tens of thousands? Hundreds of thousands?
  • How much teacher time, energy, and good will are we wasting with all of this duplicated effort?
  • How many more teachers would use an effective option that had already been vetted by another teacher if they knew about it and didn’t have to re-invent it from scratch or take the risk of "being first"?
  • Looking beyond first grade math, how many students would benefit if every teacher didn’t have to reinvent the wheel like this on every big and little problem they encounter in the classroom? Presumably every single student. That includes your children, and mine.
Here’s the punch line:
This is going on all the time.  Millions of such private experiments are presumably being conducted by teachers every day in classrooms across the country and around the globe. The only thing that’s missing to capture the value of that activity is a bit of systematic record keeping and a way to share results.

Where do we go from here?

I hope that helps clarify what I meant when I said we need to stop arguing about what people think will work in education and start experimenting to determine what actually works

In fact, I now realize that I mis-spoke when I first said that. We don't need to start experimenting - that part is already happening. As a first step we simply need to start sharing what teachers have discovered about what works and doesn't work as they try – sometimes desperately and at significant personal cost – to give our children more of what they need to succeed in school and in life.

5 comments:

  1. I’ve read your last three posts with interest. You, and those who replied, raised many important questions. I’ve been mulling these over wanting to reply, but did not know exactly where I wanted to weigh in first. I think this is a good spot.

    I can relate to the two teachers you cited as I am a first grade teacher and I too wrestled with why my students struggled with number sense (as well as issues of curriculum and pacing). Independently, I followed (basically) the outlines you gave as examples of how scientific experimentation happens every day in real time. I even attempted to replicate this investigation with our kindergarten classes, but was unable to do so for a number of reasons. So, I am one teacher, with one positive experience, and no real way to systematically analyze and disseminate what I learned so that others may benefit.

    I believe there are solutions to this problem, just as there are solutions to most problems, but we have not hit upon them as yet. Or have we? Some educational content areas do a better job than others when it comes to scientific experimentation, sharing of results, and replication: elementary literacy for example. Math, it seems, is the content area that has received the least amount of real time classroom research.

    I agree sharing is a first critical step. Unfortunately, many times this sharing does not even happen between grade level teams, much less across districts and outward. I wonder why? I think the answer is complex and we need to spend some time looking at this to determine the multiple (possible) reasons, and multiple contexts in which this occurs, discuss possible solutions, find models where sharing works, and then test each of these. Sounds strangely familiar! It might seem like we would end up back at square one with the question of how do we share results, but the idea is that our experimentation would have led us to that (those) answers(s).

    So, let’s start with this: where have you found actual models of sharing that work? (By “work” I mean information is diffused at scale, models are replicated and tested, and the cycle continues.) Let’s use this space as a sort of huge white board where we can brainstorm ideas, “write” them down, cross them off, try again...and actually, actively, work to find solutions.

    ReplyDelete
    Replies
    1. Rene,
      Thanks for sharing your experience. I feel strongly that we should support teachers like you in doing that, and that it shouldn't be nearly as difficult as it is.

      > Unfortunately, many times this sharing does not even happen between grade level teams, much less across districts and outward.

      Yes - this is somewhat puzzling. Teachers in other countries, like Singapore and Japan, share much more and routinely produce better outcomes. (So these are two places we can find successful models of teacher sharing.) One challenge is that the conditions are different across countries. In many other countries, teachers have a lighter teaching load than U.S. teachers because there is a recognition that they need more time for preparation during the school day. That makes it easier to collaborate with peers.

      Maybe we can start by looking there. When do you prep, Rene? How does your prep schedule match up with the other teachers in your school, district, extended network? What times throughout the school day or week would you have time to share ideas, insights, and questions with your colleagues if everyone were inclined to do that?

      Delete
    2. P.S. LOVE the "huge whiteboard" idea!

      Delete
  2. It seems to me that sharing the results of these "experiments" would be wonderful. It also seems that this is a problem that happens throughout society, not just in schools. How many employees figure out better ways to solve work issues but this solution never gets disseminated to other employees?

    Moreover, if each teacher is running experiments, wouldn't it be difficult to figure out which experiment had the best long term effect? By 4th grade the student would have been involved in 5 different teacher's experiments. Where is the control?

    ReplyDelete
    Replies
    1. Great point, Jeff.

      In the two examples cited in the post above, one decent control is the previous year's standardized test scores. Disappointing scores are frequently the reason teachers start looking for better tools in the first place. It's not a perfect control, but then again we don't need perfect (can't afford it, in fact). We just need to keep getting better over time.

      Delete