How Can Education Experiments Make Education Better?

In a previous post, I suggested that to make education better we should “Stop arguing about what people think will work in education and start experimenting to determine what actually works.”

There were a number of reader comments (thanks to Jeff, Stacie, and Suzanne in particular) that helped me to understand that the word “experiment” may carry some extra baggage that wasn't part of what I had in mind here.  In this post, I’ll try to clarify what I mean and offer an example.

How not to run education experiments

First, a few things I don’t mean when I talk about “education experiments”:

• I don’t mean we should bring scientists in white lab coats into the classroom so that they can better understand how our children’s brains work – that is cognitive science, not education science.
• I don’t mean we should bring academics in white lab coats into the classroom to test their speculative or radical new math curriculum for the very first time on our children for the benefit of future generations.

Don’t get me wrong – I think both basic and applied academic research are extremely important and have their place (with appropriate safeguards and informed consent all around). In particular, these kinds of research can benefit society greatly over the long term. But academic research is not what I had in mind here for two reasons:

1. I am more interested in research that has a good chance of benefiting the participating students in the short term, and
2. I am interested in research that leverages the teacher’s special knowledge and insight about the capabilities and needs of those particular students.

A different take on education experiments (with an example)

What I am proposing is more akin to action research, or design-based research.  To clarify what I have in mind, I’ll give an example from real life.

In my work as an educational designer and consultant, I have the pleasure of talking with some extremely dedicated teachers. Sometimes I meet them when they send a message, out of the blue, with a specific idea or question about how to do more for their students. Below is an example of one such message I received about a year ago.

Key points to note about the situation:

• This teacher has identified a problem: Something isn’t working right, and my students are struggling in math.
• This teacher has a hypothesis about the root cause of the problem: I believe some of my students lack number sense, which prevents them from understanding the more complex material in the math curriculum.
• This teacher has done some research (most likely on her own time) and identified an option that she believes could address the root cause better than the current curriculum alone: I have found an iPad math curriculum that I believe will solve the problem by developing my students’ number sense.
• This teacher has encountered a barrier that prevents her from trying the option: I don’t have all the technology I need (wifi) to use the app in my classroom.
• This teacher is actively seeking creative ways to overcome the barrier: Is there a way I can use the app-based curriculum without wifi? (In my experience, teachers are willing to go to great lengths to make something work if they believe it will help their students.)

This situation has all the trappings of a good old-fashioned experiment. What happens next?

Here are two common outcomes:

• Either: The teacher gets no support to overcome the final barrier, she can’t even try out the option she believes will help her students, and her students continue to struggle with math – probably for life (because math misunderstanding – like math understanding – is cumulative). Sadly, this is probably what happens to proactive teachers and their students most of the time. The teacher takes the ball to the 99 yard line and for any number of reasons can’t carry it the final yard alone so it sits there. The students miss out. The dedicated teacher gets a little more burnt out – maybe this is even the last straw and she leaves the profession. Everyone loses. 
• Or: The teacher finds a creative way around the barrier, tries out the math app with her students this year, and based on her results she decides whether to use it again (or try something else) with her class next year.  The teacher goes to great lengths to get her students more of what they need. Her twenty-five students benefit. She feels empowered - maybe she can make a difference through diligence, resourcefulness, and a lot of hard work. But…did her idea work? How well? What’s the evidence that it did or didn't? Did it work for some kids and not others? If so, which ones? What did she do to make it work? That teacher has a lot of valuable, actionable insight as a result of her experiment, but it was a private experiment and so the world will never know about it.

The answers to those last questions would have been handy six months later when I received this message from another teacher:

It’s déjà vu all over again.

Key points to note about this situation:

• This teacher has independently identified the same problem: Something isn’t working right, and my students are struggling in math.
• This teacher has independently produced the same hypothesis about the root cause of the problem: I believe some of my students lack number sense, which prevents them from understanding the more complex material in the math curriculum.
• This teacher has done some research (most likely on her own time) and independently identified the same option that she believes could address the root cause better than the current curriculum alone: I have found an iPad math curriculum that I believe will solve the problem by developing my students’ number sense.
• This teacher has encountered a slightly different barrier that prevents her from trying the option: I don’t have all the technology I need (iPads) to use the app in my classroom.
• This teacher is also actively seeking creative ways to overcome the barrier: Is there a way I can adapt the app to a whole-class format so I can make it work with the technology I do have available?

The teacher from California observed a pattern, had a hypothesis, came up with a test of that hypothesis, and – if she was persistent and lucky enough – was able to run the experiment.  But no one except perhaps a few colleagues will ever hear about it, whether it was successful or not. As a result, the teacher in Tennessee has to do all of the same work over again, from scratch, without guidance or data to help her make informed decisions. Which raises a few questions:

• If these two teachers face the exact same problem with teaching math, how many others are there? Tens of thousands? Hundreds of thousands?
• How much teacher time, energy, and good will are we wasting with all of this duplicated effort?
• How many more teachers would use an effective option that had already been vetted by another teacher if they knew about it and didn’t have to re-invent it from scratch or take the risk of "being first"?
• Looking beyond first grade math, how many students would benefit if every teacher didn’t have to reinvent the wheel like this on every big and little problem they encounter in the classroom? Presumably every single student. That includes your children, and mine.

Here’s the punch line:
This is going on all the time.  Millions of such private experiments are presumably being conducted by teachers every day in classrooms across the country and around the globe. The only thing that’s missing to capture the value of that activity is a bit of systematic record keeping and a way to share results.

Where do we go from here?

I hope that helps clarify what I meant when I said we need to stop arguing about what people think will work in education and start experimenting to determine what actually works. 

In fact, I now realize that I mis-spoke when I first said that. We don't need to start experimenting - that part is already happening. As a first step we simply need to start sharing what teachers have discovered about what works and doesn't work as they try – sometimes desperately and at significant personal cost – to give our children more of what they need to succeed in school and in life.

Can We Make Education Better, or Is This Really As Good As It Gets?

I attended a summer conference years ago at Brown University. The conference theme had to do with making brain research more useful in practice. I was a doctoral student at the time, working in the emerging domain of Educational Neuroscience. The field was so new it wasn't even established yet, it was starting to generate some buzz, and this was one of the first conferences of its kind. I was pretty excited to be there.

One talk was by a cognitive neuroscientist renowned for his brain imaging studies of memory and perception (how we see, hear, etc.). This particular presentation, though, was about a study he was doing on learning. Learning and motivation have been my primary interests for a long time, and so I was keenly interested in the talk. During the Q&A following the presentation I asked, "Do you have ideas about how this kind of brain research might help us make education better?"

Based on prior experience, I expected one of two answers.

Either:
"Yes, I have several ideas about how this might be useful down the road.  For example..."

Or:
"No, this laboratory work is too preliminary and too far removed from educational practice for me to speculate about that."

His actual answer was far more interesting (and surprising).  He said:
"I think we do pretty well in education already. I don't really think it's possible to do much better."

I was taken aback. First, he dismissed my question instead of answering it. But, more importantly – could he really be suggesting that education is about as good as it can possibly be? And that a deeper understanding of the biology and psychology of learning won't lead to better education? At the time it hadn't occurred to me that people - especially people who study learning - could seriously hold that view.

But he got me thinking…

Why should anyone believe that education could be better than it is?  And how much room for improvement is there, really?

These are quite important questions.  After all, if the best-case scenario would be, say, 10% improvement over what we do now, then it probably wouldn’t be worth the extra cost.  But if, on the other hand, we could double the return on our educational investment (100% improvement), then that would really be something to get excited about!

How much more efficient could education be?
First, let’s think about educational efficiency.  One way to determine that is to compare how long it takes students to master a specific amount of content under different instructional designs. One telling example comes from the SHERLOCK Project, in which cognitive scientists demonstrated that they could replace four years of on-the-job training (in troubleshooting electronics) with 20-25 hours of computer-based training to produce the same expert-level performance. Four years of training compacted into less than three days through the systematic application of Learning Science.  The authors of this article point out that “SHERLOCK achieves this stunning result in two ways, by affording the opportunity for extensive practice and by creating educationally effective instructional conditions” (p. 9).

So, based on this example, we can estimate that in at least some domains education could be made not just 10% or even 100% more efficient than it currently is through optimal instructional design, but on the order of 100x – or 10,000% – more efficient!

Wow. I’d say that puts the score at:
Me: 1
Cognitive Neuroscientist: 0

How much more effective could education be?
Now let’s think about educational effectiveness. One way to determine that is to compare levels of student achievement in a subject area like math resulting from different instructional designs. The cognitive scientists behind the RightStart project did just that.  In one study, they followed three groups of children for about five years (from preschool through third grade), measuring their math achievement each year.

• Group 1 was a high-resource group – children from middle- and upper-income homes who attended a magnet school with an enriched math curriculum (not specifically informed by Cognitive Science).
• Group 2 was a high-ability group – children from lower-income homes who were identified as having superior math achievement when they arrived at school. They received no special or enriched math instruction.
• Group 3 was the experimental (RightStart) group – typical children from lower-income homes with no special aptitude for math.  In addition to their regular school curriculum, these children received about 20 hours of intensive supplementary instruction in “number sense” (the conceptual foundations of math).  The special number sense curriculum was explicitly informed by Cognitive Science.

The RightStart group started out with the lowest math achievement of the three.  Over the course of five years, however, they reached higher levels of achievement than either the high-resource or the high-ability group.  Evidently, optimal instructional design based systematically on Learning Science can take the children who would normally sit at the bottom of the class and reliably put them beyond what is currently the top of the class – and they’ll stay there for years.

Wow again.  I’d say that brings the score to:
Me: 2
Cognitive neuroscientist: 0

Cognitive Neuroscience is not Education Science
Based on available evidence, it seems the Cognitive Neuroscientist was just plain wrong in saying that education today is about as good as it gets. But he’s a senior scientist studying learning – so where’s the disconnect?

I think Clarke’s first law applies here:
When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

What I have further come to understand in the decades since (thanks in no small part to this book by Dewey) is that there is a vast difference between a Cognitive Neuroscientist who studies learning and an Education Scientist who studies education. Much could be said about the differences between them. For now, I’ll simply point out that the former apply systematic methods of inquiry to describe neurological and psychological processes, while the latter apply systematic methods of inquiry to improve educational processes. The difference is analogous to that between the chemical sciences and the medical sciences.

I have spent much of the past two decades helping people produce the learning outcomes they desire (education) by applying systematic methods of inquiry (science). I am an Education Scientist. In retrospect, I think the Cognitive Neuroscientist was not wrong to dismiss my question. I think I was wrong to ask it of him - in the same way you wouldn't ask a chemist (even a brilliant one) to prescribe your medication. That's simply not his area of expertise.

Long experience in the field supported by evidence from programs like SHERLOCK and RightStart have led me to believe that we can make education better across the board. Not a little better - a whole lot better. Imagine, for example, that our schools could produce twice the learning in half the time at a fraction of current cost. I believe this is feasible today, in existing schools, with our current students, teachers, and facilities.

The way we can make education substantially better is with a straightforward three-step process. This process builds on the one applied successfully by the scientists behind both SHERLOCK and RightStart:

• Base educational designs on scientific models of human learning.  There's over a century's worth of relevant research (including work in Cognitive Neuroscience) that is hardly being used in educational practice. We need to start using it. 
• Use technology to dramatically expand access to high-quality learning experiences while lowering cost. Today, we could basically create for every learner a "personal-tutor-in-a-box." Students could work at their own pace on a personalized curriculum. The system could collect rich data on student performance. Teachers could spend more time actually teaching.
• Stop arguing about what people think will work in education and start experimenting observing, acting, and sharing more systematically to determine what actually works. Simply apply scientific thinking to educational processes the same way we do to every other major domain of human endeavor like engineering, medicine, agriculture, and economics. 

Wash. Rinse. Repeat.

Today, we have all of the components we need to make education much better. There is definitely a way. The question is, do we have the will to make it happen?

I Believe We Can Make Education Much, Much Better Than It Is Today

Welcome to The Education Scientist. The purpose of this blog is to provide a forum for discussing how we can make education systematically better – especially in the short term – through methods proven to work in other applied domains like medicine and agriculture. 

I value openness and transparency. This first post is intended to tell you where I stand and where I am headed so you can decide if you are interested in walking with me for a while. The summary below is a 50,000 foot overview of the landscape. I plan to expand on each of these points in future posts.

Here, in a nutshell, is my point of departure:

• I believe that learning may well be the ultimate Good.
• I believe that the purpose of formal education systems (like schools) is to facilitate specific learning outcomes desired by a community.
• I believe that compulsory education can be a powerful public Good, but that it brings with it certain moral obligations on the part of those who provide it.
• I believe that our systems of education (public and private alike) produce less than 50% of the learning that even the average student is capable of. In particular, I believe that we have the capability to produce twice the learning we are currently producing in half the time at a fraction of current cost.
• I believe that if we chose to we could accomplish this today, for the average student, in our existing public and private educational institutions.
• I believe that the stakes in this area are very, very high. In particular, children’s life courses are being determined powerfully by the quality of their education. The fate of entire societies will in turn be determined by their citizens’ life courses.
• Because the stakes are what they are, I believe that knowing how to provide better education for every child creates a moral obligation to do so – for every child – and to act with all possible haste. Every year another cohort of children – our children, grandchildren, nieces, nephews, and friends – advances through the system without getting what they need to survive and thrive in the world we have created for them.

Education Science is the systematic application of methods of inquiry to drive the educational outcomes that a society desires. I believe that Education Science can enable us to produce dramatically better learning over time, in the same way that Medical Science, Agricultural Science, and Political Science enable us to produce (respectively) better health, global nutrition, and political stability over time.

I believe it is within our capability to make education much, much better than it is today. If you believe this too, then I hope you will walk with me a while by joining in the conversation here.