Minecraft Scientists Ep. #1: Fishin' In the Rain (STEM Education)

Minecraft might be the ultimate tool for STEM education (Science, Technology, Engineering, and Mathematics).  Check it out...

(Want to try the experiment yourself? Get instructions in the spreadsheet linked below.)

What Is Science?
Science – experimental science in particular – is like a game of 20 questions we play with Mother Nature. We have a question like “Am I more likely to catch fish on rainy days than on clear days (or does it make a difference)?” Maybe no human being knows the answer, but Mother Nature does. 

Only she isn’t going to just tell us the answer straight up – she doesn’t speak English, after all. We have to be more clever in how we get the answers from her. 

The Scientific Method
One thing we can do is make a guess (form a hypothesis), and then submit the guess to her (run an experiment).  For example, for the question about when you are likely to catch more fish, we might run an experiment like this: go to the same exact fishing spot each day for 10 clear days and 10 rainy days, always arriving and leaving at the same time of day, always using the same rod and tackle, and always using the same bait. (We call this “controlling” for location, time of day, total fishing time, equipment, and bait, and we do it to make sure that it’s the weather and not some other factor like the particular bait we use that is causing any differences in how many fish we catch each day.)

The data we get from the experiment (the number of fish we catch on 10 clear days and 10 rainy days) is Mother Nature’s answer to one of our questions, and it might range from “yes” to “probably yes” to “maybe” to “probably no” to “no” to “no comment.”  (Also: we usually have to analyze the raw data in some way to find out what her answer actually is.) Through playing this game of 20 questions with Mother Nature and pulling together all of the clues we get from our experiments we can get more precise and certain answers to our questions over time. 

21st Century Science Education: Real Science in a Virtual World
The best (and most fun) way of really learning science is by doing real experiments like the 20-day fishing experiment described above. But real experiments like this one often require a lot of money and time, and so we can’t really do that for every kid.  

Or can we? The video shows an example of doing real science in the virtual world of Minecraft a lot faster and less expensively than we could in the real world - and without having to get wet! 

File this under “things I wish they had when we were in school.” 

Science Can Surprise Us!
Swifty7777 and I have run the experiment once (collecting 20 Minecraft days’ worth of data between the two of us – less than an hour of real time) – and, as often happens in science, we were surprised by what we found.  In fact, our results seem to contradict what the Minecraft wiki says is true - they indicate you should catch about 20% more fish in the rain than on a clear day. (Watch the video for details on what we found.)

Who is right – us or the Minecraft wiki?

Your Turn! Try the Experiment Yourself
You can help us figure it out: replicate our experiment (that means run it again the same way – just like real scientists do!) and see if you get the same results. 

Besides, if you really want to learn science, you have to do more than watch videos of other people doing it - you have to get in there and work through it yourself.  We've provided an Excel spreadsheet for you to download here that makes it easy for you to do that.

If you email your filled out spreadsheet to info@i4kd.com with the subject line “Minecraft Fishing Experiment #1”, we’ll post your results so people can compare them to ours.

Happy fishing!

Epic_MC_Player and Swifty7777
// The Minecraft Scientists


Instructions for running the experiment yourself are here (same as link above).

If you liked this "learning with Minecraft" video, you might also like these:
Minecraft Math #1: Numbers - Even, Odd, Prime & Square Root
Minecraft Math #2: Addition, Multiplication & Commutativity

Common Core Math Standards Making You Crazy? Some Things to Consider

Pop Quiz!

(No Googling or peeking at other people's answers before responding, please - this is a closed-book quiz. Also: this will not go on your permanent record.)

With reference to the long multiplication problem above, please answer the following questions.

Q1) What does the "4" that is circled mean?

Q2) What does that zero in the second row mean (the one that is circled)?

We'll come back to this later.

Common Core Math Standards: What's the Point?

Our topic today is the Common Core Math Standards.  They seem to have some people in a tizzy.

The picture at left, for example, has been making the rounds on the internet. Evidently someone snapped this picture of their child's math homework because they were enraged by it, and lots of people are hopping on the bandwagon.

This is curious to me.

The intent here should be pretty clear - the point is to develop the child's conceptual understanding of subtraction. This happens in parallel to developing their fluency in the standard subtraction algorithm that we all learned as children (not shown on this page, but also covered in the Common Core standards).  The standard subtraction algorithm is efficient for calculating, but doesn't support understanding.

Do We Really Need to Change the Way We Teach Math?

Why would we want to teach a child to understand the concepts behind the algorithms?  Isn't that just a waste of time?  We all did fine just learning the algorithms by rote, didn't we?

Well, no.  It turns out adults in the U.S. aren't very good at math.

Speaking of which - let's take a moment to reflect on the quiz above.  How did you do on it? How quickly could you answer? Did you step away to search the internet for some clues or look at the poll results before submitting your own answers? (Naughty monkey!) How confident were you in your responses?

The simple fact is we are failing our kids in math education, and we have been for generations.  Here are some more fun facts to illustrate the point:

• According to the National Assessment of Educational Progress, only about two in five U.S. 4th graders are proficient in math.  That drops to about one in three students by 8th grade.
• According to the Programme for International Student Assessment, the U.S. ranks below the international average on tests of mathematics.  This is despite the fact that the U.S. spends more per pupil than other countries.

It's pretty clear we're doing something wrong. Solving big problems like this requires big changes.

How Can We Do Better?

If we look at some of the top performing countries, like Singapore, and ask what they are doing that we aren't, the most obvious difference is that they focus on conceptual understanding a whole lot more than we do, and they spend a lot less time teaching things by rote.  They teach the kids several models and methods for thinking about numbers and operations on numbers, for example - methods like the "Counting Up" method represented in the picture above. It's not a secret. We have just stubbornly refused to do it (until recently, anyway, with the arrival of the Common Core math standards).

But Aren't We Just Complicating Things?

Teaching concepts may take more time than memorizing a few recipes for calculating without understanding (at least initially), and some people seem to object to spending the extra time.  

This is extremely short-sighted.  There may be more time spent initially developing understanding, but that investment will pay dividends many times over across the student's time in school, from kindergarten through high school (and beyond).  Understanding the concepts makes later learning far more efficient - and effective. By spending more time developing foundational understanding, we could actually get better outcomes while spending less time on the math curriculum overall. Not only that, but students who understand what's going on have a much better experience, are more engaged, and are more confident. All good.

Why Can't We Just Do What We've Always Done?

What does it look like when we fail to teach children conceptual understanding?  The video below shows one example of a second grader working on some grade-level math problems.

Notice that she knows the numbers and she knows how to count - these are typically learned by rote. Her conceptual understanding is very weak, however, and as a result she has to go through a laborious process of counting up from zero to answer simple questions like "How much do I have if I add 10 to 35?"  And then she gets the wrong answer. Repeatedly.

Time invested in developing authentic understanding is not a waste of time.  Quite the contrary. The real waste is time spent teaching without developing understanding - which produces the kind of disjointed, brittle, and tentative knowledge shown in the video - which is ultimately quite useless and will likely fade away rapidly.

This is all too common.  This could well be any of our children.  The maddening thing is that there's every reason to believe this child - and virtually every child - is completely capable of understanding the concepts she would need to reason fluently about the questions being asked of her in this video. She's not failing at math. Our education system is failing her. The same way it has failed generations before her.  In large part by teaching math by rote, without conceptual understanding. (Don't scapegoat the teachers, by the way - the root problems here are systemic.)

We could try to press ahead as some people are advocating and just teach this child the algorithms for long addition, multiplication, and division as we have always done.  But with such a weak foundation of understanding, what would we really expect to achieve that way? Her performance on those would quickly come to look like her performance here - labored, uncertain, and error-prone. Eventually she could well stop using the algorithms for lack of confidence, or even forget them altogether.

This is the sort of large scale, systemic problem the Common Core Math Standards are meant to rectify.

Could the explanation in the textbook shown in the picture above be edited for clarity?  Sure it could.

Is that evidence that the Common Core is a failure and should be trashed?  Far from it. We've been doing math education wrong for a very long time. The Common Core Math Standards represent a big step in the right direction - in the direction of what Singapore and other top-performing countries do, in fact.

Some people seem to think the textbook image above is crazy. What's really crazy is recognizing that the status quo is not acceptable while repeating the same educational processes generation after generation and expecting a better result.

Come to think of it, wasn't that literally Einstein's definition of insanity?

Celebrating Howard Gardner's Extraordinary Mind, Life, and Work (with a free book download)

Howard Gardner is the Hobbs Professor of Cognition and Education at the Harvard Graduate School of Education.


Howard Gardner has been identified by Foreign Policy and Prospect magazines as one of the 100 most important public intellectuals in the world today.  His work has fundamentally changed the way many people (and institutions) think about intelligence (with his theory of Multiple Intelligences), creativity, and education (to name just a few of the areas he has touched).

Happy Birthday, Howard!

Howard turned 70 last year.  To celebrate, his wife (Ellen Winner) and one of his former students (Mindy Kornhaber) hosted a Festschrift in his honor.  (A Festschrift – derived from the German for ‘celebration writing’ – is a collection of writings published in honor of a scholar.) They invited Howard’s teachers, peers, colleagues, and former students to contribute essays inspired by his work and his relationships with them.  One hundred and sixteen of Howard’s close colleagues contributed to the two-volume work, entitled Mind, Work, and Life: A Festschrift on the Occasion of Howard Gardner’s 70th Birthday. Each contribution includes a personal note from the contributor and a personal response from Howard.  Running 605 pages in length, this is quite a remarkable work, providing a unique and intimate portrait of this extraordinary man and his profound influence on some of the people who have worked most closely with him.

Mind, Life, and Work: A Festschrift on the Occasion of Howard Gardner's 70th Birthday

The complete two-volume Festschrift is available for free download as a PDF here, or if you prefer physical books you can buy it at cost from Amazon here.  Other options, including kindle versions of the two volumes, are listed here.

My contribution (starting on p. 223) is entitled, There’s More Than One Way to Bridge a Gap: On the Promise of Computational Neuroscience for Education.  I wrote it as a doctoral student in Education, soon after I took my first class with Howard.  At the time, I was just beginning to wrestle in earnest with the question: “How can scientific insights about the brain and mind help us make education better?”  As reflected in this essay, Howard’s teaching was instrumental in helping me frame the key issues in a new and more productive way, which I have continued to build on to this day.  If you are interested in the relationship between the brain and mind, or in how we might go about leveraging insights about the biology of learning to improve educational practice, you might find it interesting.  I look forward to reading your comments on that or anything else in the book.

Enjoy!

Teacher Appreciation Week: A Personal Tribute to Teachers

It’s easy to see and criticize the flaws in our education system here in the U.S. – especially our public education system. And teachers are the face of public education – which means they often take the brunt of the criticism and discontent from all quarters, even for issues completely beyond their control.

Teacher Appreciation Week is a time to pause and reflect on the good and important work that teachers do – and to openly express our gratitude to these people who have dedicated their lives to helping our children (and us, when we were children ourselves) become happy and productive members of society.

I am a product of public education. I can only imagine what my life would have been like if I had been born into a society without it. And I can only imagine what my school experience – and my life – would have been like without the benefit of the extraordinary courage, kindness, and skillful teaching exhibited by many individual teachers I had the good fortune to meet along the way.

Carole Rosen-Kaplan, for example, was my 11th grade English teacher. She became a dear friend. Sadly, she passed away recently. Carole’s sons asked me to provide some comments for her memorial service. I realize now – too late – that although we talked a couple of times a year, I never told her how much I appreciated her as a teacher, or what a profound impact her teaching had on my life. I regret that.

I’m trying to make up for it a bit by “paying it forward.” Teaching is extraordinarily hard work. Often it’s thankless. Sharing the comments below is my way of saying to all of you, teachers:
Thank you.
I appreciate what you do.
The work you do influences and transforms your students’ lives in ways you (and they) will probably never know.

 ***

I met Carole when I was a high school junior. Her English Composition class wasn’t the English elective I wanted that year, but it was the only one that fit my schedule.

In retrospect, the class was so thoughtfully crafted and compelling that even today (decades later) I could probably write down most of the syllabus from memory.

I wrote my first short story for that class. It was about an archaeologist who stumbles upon a powerful relic in Egypt and uses it to travel back in time but ends up trapped. Carole (Mrs. Rosen-Kaplan to me then) asked me if I plagiarized it. I thought “Wow, that must be a pretty good story.” I don’t think it was her intent in that moment, but that honest exchange started me thinking that if I could accidentally make an English Composition teacher believe I had stolen a published story then maybe I could be a real writer who actually published stories. (Inspiration takes many forms.)

I wrote several chapters of a novel that year (which she read and commented on in her own time), and later became president of the school’s Creative Writing Club - my first formal leadership role. One of my short stories won a prize in a writing competition. I later wrote a short story for my college essay. It worked! (I got in.) AND a love poem for a woman. It also worked! (She’s my wife.)

It turns out that English Composition is crazy powerful stuff.

That year I argued against nuclear stockpiling in a mock trial of the global superpowers and wrote passionately about the Vietnam war, the Holocaust, and human slavery.

The class *involved* writing but it wasn’t *about* writing. It was about love and hate, good and evil, right and wrong, politics and power.

As I reflect back on the class now, though, I realize that what is remarkable to me about Carole’s teaching is that we didn’t simply *read* other people’s thoughts on these themes or even write about them in the abstract. When writing for Carole we had to choose sides - we had to “try on” different points of view and in the end commit to one. On the theme of War: will you, as the author of this essay, choose to glorify or vilify it?

More importantly - on the subject of war: will you, as the author of your own life, choose to glorify or vilify it?

But helping us find our voice was only one of Carole’s objectives. Her other objective was to impress upon us our responsibility to use it.

Perhaps the most poignant statement of individual responsibility I have ever read is captured in these lines by Martin Niemöller (assigned in Carole's class) about the Nazi purges:

First they came for the Socialists, and I did not speak out – Because I was not a Socialist. 
Then they came for the Trade Unionists, and I did not speak out – Because I was not a Trade Unionist. 
Then they came for the Jews, and I did not speak out – Because I was not a Jew. 
Then they came for me – and there was no one left to speak for me. 

I have recalled this passage to mind many times over the decades. It reminds me of my right - and my responsibility - to use my voice. More importantly, it reminds me that sometimes doing the right thing is deeply uncomfortable all around - and that courage does not mean the absence of fear but rather doing the right thing in the face of fear. Speaking up and speaking out is not the responsibility of a chosen few - it is the right and responsibility of every human being.

Carole’s class was called “English Composition” but the full title should have been something like English Composition: How to Find Your Voice and Raise Hell Through Writing.

I think Carole saw promise in me as her 11th grade English Composition student and was disappointed that in the end she didn’t inspire me to pursue a creative writing career.

But here is what I would say to her in response to that…

Carole,
Thanks in part to you I know who I am and who I aspire to become. I know what I stand for. I stand for what is right, what is true, what is just, and what is good. I stand for people - especially the people who can’t stand for themselves, like children. I stand for the right of every human being to discover their own voice and have the opportunity to be heard. I stand for the importance of helping people discover who they are, who they want to become, what they are deeply passionate about, and how to become the authors of their own lives.

No, you didn’t inspire me to become a creative writer.

You inspired me to become an Educator.  Like you.

All my love,
-Mike Connell
April 26, 2014

Mind the Quicksand: A Word of Warning to EdTech Investors

I read about a new EdTech startup this morning in TechCrunch. It's called Galxyz. I started writing a comment on the article page itself, but then it got really long and kind of "meta." So I thought it might make an interesting blog post.

Here's the description of the company from TechCrunch:

Galxyz (it’s pronounced “galaxies”, and it’s a nightmare to spell) is building a science-focused game for tablets and smartphones. And the company really is just focused on one game — Rashid [the founder] described it as “an intergalactic science adventure,” one that kids could potentially play for years, battling a villain called King Dullard across the galaxies. As they do so, they’re also learning about science at their own pace.

You can check out their epic promo video here (I would embed it, but that might violate copyright).

Sounds interesting enough.  Although an antagonist called "King Dullard" is a little on-the-nose, if you know what I mean.

What's the new angle on education, I wondered?

Apparently the idea came to Rashid as he saw his own children playing educational games, which he found lacking in several ways. He said they weren’t engaging enough, the content wasn’t deep enough, or they required the parents to get involved in order for the kids to advance. That second point is why he’s focusing on a single title — so that kids can just keep playing rather than running out of material after a few weeks.

Wow - it seems so obvious once he points it out.  I wonder why this hasn't occurred to anyone before? 

Anyone who follows this blog knows that I believe it is entirely possible to make education orders of magnitude better.  And that I applaud any authentic effort to try.

But from where I stand, this has "Titanic" written all over it.  

I am confident that producing truly effective education at scale is harder than any business or engineering challenge the good people at Galxyz have ever faced.  

This isn't about them, though.  I wish them well.  For purposes of this blog post, they represent just the latest in an endless parade of investor-backed EdTech entrepreneurs who seem to have almost exactly the same creation story.  I'm not upset about it.  I'm more intrigued.

One aspect of education that fascinates me is that to outsiders it always looks so *simple*.

As simple as running across that wet, sandy patch of ground in the jungle to get to a fabulous treasure twenty feet away.  But wait...it's hard to believe no one has run over there and taken that treasure before...hmmm...don't you wonder why? (Hint: That's not sand.)

What follows is my prediction for this venture.  

To be clear, this is not what I wish for them.  I want truly better education more than just about anything, and I'm indifferent as to whether it comes from a for-profit or non-profit venture, as long as we get it.  And it's so close I can almost taste it.  

The prediction is based on a pattern that I see play out over and over and over and over and over again (that's right - five 'overs').  

It looks like this...

Day 1: Outlook bright, mattress so overflowing with cash I can't climb onto it so am sleeping on the couch, burn rate 7, feeling great, gonna make history and be a hero by fixing education because I'm...
(Choose all that apply)

• A product of an education system, and therefore I know how to educate...more better...than anyone
• More funner than anyone in education today
• In possession of more revolutionary technology than those other people
• More smarter
• Mo' richer
• A bit confused about the difference between money, intelligence, and expertise

Day 30: Epic teaser promo video produced, first prototype created and friends (who are also employees, or hoping to be) are saying it's definitely the next big thing - time to pick up the pace of hiring animators, writers, and engineers!  Mattress so overstuffed with clams that I rolled off it last night.  (Ouch.)  Outlook blazing, increasing burn rate to 9.0.

Day 180: A bunch more concepts storyboarded and prototyped, things seem to be moving right along.  Still, things are getting a bit confusing.  We discovered that narratives are inherently linear and learning appears to be inherently nonlinear, so that's creating some "design challenges" (so-called).  Can the kids go through the same narrative ten times if that's what they need in order to understand the concept?  But wait - even if they will tolerate that, isn't one definition of insanity "doing the same thing over and over while expecting a different result"?  Should we have multiple narratives for each concept? Wow - that would be expensive.  Let's stick with the one narrative.  Should we at least present each concept in multiple different ways to help kids understand it? But will all those choices confuse them?  And what other ways are there to present the concepts within a single narrative? This is taking longer than we thought - better hire more animators and engineers!  Mattress slightly overstuffed with cabbage. Burn rate goes to 11. Outlook bright.

Day 240: Somehow a couple of teachers got wind of what we were doing and we invited them in for a demo.  They were really harshing on our mellow.  They think the whimsical five minute cut scenes between short sets of learning activities are too long - complaining about how that's going to "steal learning time" or something.  But don't they understand that's what makes learning fun?!  (And how many simoleons we have invested in those videos?!)  They also wanted to know what kinds of student performance data we are going to provide.  We showed them the awesome score meter and the leaderboards but they didn't seem to get it.  That's why we didn't want to bring any teachers up in here - we just knew they wouldn't understand our Vision.  Mattress still stuffed with ample Benjamins. Burn rate holding steady at 11.

Day 330: Brought some kids in to play test for the first time.  They thought the narrative was kind of hokey and felt tacked on to the learning activities.  Lenny's kid called it "chocolate covered broccoli."  That's why I didn't want to bring students up in here - we just knew they wouldn't understand our Vision. The writers and animators are getting a little up in arms because people keep changing requirements and trying to mess with their narrative.  I sympathize with them - the narrative is the hard and expensive part, so shouldn't we revise the science material to fit it instead of the other way around?  Running a bit behind schedule.  Mattress feeling a bit lumpy.  Better pull back on contractors to conserve cash.  Burn rate reduced to 8.

You can imagine where it goes from here - the way of 38 Studios or a quick exit that amounts to a soft crash landing.  Like I said - quicksand.  The pattern is that they don't discover where the real complexities in education lie until they are in it up to their necks.

Here's some unsolicited advice for engineers and investors who are eyeing EdTech:

• If you think you can crack the code on better education with money, you are wrong
• If you think you can crack the code on better education with raw intelligence, you are wrong
• If you think the core challenges in EdTech are technical, you are wrong
• If you think the silver bullet is in "making learning fun" or "engaging" students, you are wrong
• If you think the solution is in making clever lessons for each concept, you are wrong
• If you think you can solve this problem with better graphics, animation, and narrative, you are wrong
• If you think bringing even all of the above elements to the table *must* be sufficient to crack the code on better education, you are still wrong - though you might be able to generate a positive ROI that way.  Or not.

Relay Ventures, Andreessen Horowitz, and Emerson Collective (the investors backing Galxyz) - you are welcome to use the above as a litmus test when evaluating EdTech pitches in the future.  (Acknowledgement as the source is always appreciated.)

Don't get me wrong: I strongly believe there is a path to better education.  This just isn't it.

I can hear the incredulous guffaws now.  We are on Day 30 and the outlook is blazing - we have an epic promo video and the prize is practically within reach!

I don't really see any point in debating the matter right now - this is a prediction, not a challenge.  Let's check back in a year and see where things stand.  I would be pleasantly surprised to be wrong.  But I wouldn't bet on it.

Khan Academy: How Does It Measure Up? (Part 2 of 2)

This article is the second in a series. In the first article, Dr. Schwartz distinguished authentic understanding from the Illusion of Understanding and introduced five principles from learning science that support the development of authentic understanding. For further reading, check out this article published in the Journal of Asynchronous Networks.

Note: The original published version of this post was longer.  We shortened it in response to reader feedback.  If you'd like to read the original version of the post, please contact the blog owner.

Dr. Marc Schwartz 
Professor of Education at the University of Texas at Arlington and Director of the Southwest Center for Mind, Brain, and Education

Dr. Michael Connell
Ed Tech Designer & Visiting Researcher at the University of Texas’ Southwest Center for Mind, Brain, and Education

Introduction

In Part 1 of this article I posed a challenge you may still be considering. If you remember the Iceberg Challenge, your goal was to decide what would happen to the water level after all the ice melted.  For many years, what nearly all my students found particularly irritating about this challenge (and me) is that I stopped providing the answer. You might be feeling that irritation too.  

If you read Part 1 and tried the challenge, are you feeling this irritation?

My goal – then or now – was not to be irritating. My goal is to use our collective experience of the iceberg challenge to clarify what we mean when we use the word understanding, so that we’re all talking about the same phenomenon in the same way with the same expectations. 

In Part II we introduce an understanding scorecard to help expose the Illusion of Understanding and in turn define what understanding means in the area of math, and finally consider what choices may be available to Khan and all educators, especially those who work online, to better support authentic understanding.

What do we mean by understanding?

At one extreme, an understanding might mean that we know something (anything) about a subject, so that we can participate in a cocktail conversation. For example, imagine a person said to you, “I think it makes a difference to coast lines if all the polar ice floating on the oceans melts, but I’m not sure how…” Would you say that person understands Archimedes Principle? 

Alternatively, would you say a person understands Archimedes Principle if they can provide a definition or use a mathematical formula to solve for a missing variable? 

At the other extreme, would you say a person understands Archimedes Principle who can:

• Recognize deeper connections between situations that seem unrelated on the surface - such as what happens to ice melting in a glass and what happens to a balloon full of oxygen released on Mars (whose atmosphere is predominantly carbon dioxide),
• Solve a variety of novel challenges like the Iceberg Challenge,
• Explain their reasoning and articulate why they believe their answers are correct across different contexts, and
• Recognize how a new concept or formula relates to what they have learned previously, so they can start using it quickly? 

These three points of view (let’s call them “low,” “medium,” and “high” understanding, respectively) map out positions along a continuum that begin to portray understanding in a richer and more complex way. We may all discover that in the past we have been holding different assumptions when using words like “understanding” (or “learning,” for that matter).

Do these three points of view frame a continuum that feels useful to you?

Using this continuum as a shared point of reference, we can ask a couple of distinct but related questions:

• What outcomes are possible? What is the highest level of understanding that students can theoretically achieve in a given subject area on a large scale in a particular formal education system, given the available resources in that system?
• What outcomes are expected? What level of student understanding should we hold the formal education system accountable for in practice?

The two questions above may seem similar but they could hardly be more different. The first question is a question of fact – there is an objective answer independent of what we believe or desire (although it might be difficult to discover that answer – more on that later). While different people might have different beliefs about the answer to the first question, at least one of them is guaranteed to be wrong. The second question is not a matter of fact – it is open-ended and requires a community decision. Different people and communities will certainly have different views about the second question and – as long as they respect the objective limits on what is possible – none of them can be considered wrong because there is no objectively right answer.

Even though the two questions are distinct, they are related. The first question (what is theoretically possible) puts a hard limit on reasonable answers to the second question (what the community demands of its educational system). Two common mistakes that people make when reasoning about education are: 

• They assume a low level of understanding is the best that can be achieved at scale in an education system, and – without checking that assumption – they decide to set a low bar for student understanding based on it.
• Conversely, they ignore the ceiling on what is theoretically possible and make impossible demands of educational institutions. 

It is not our aim here to argue in favor of or against any particular purpose of education. What is important right now is to know what we mean when we say we want students to understand _______ (and you fill in the blank), and to be clear about which question we are discussing at any given time (what’s possible vs. what's expected).

How do we determine what level of understanding is possible?

Formal education systems are so complex that it is difficult to analyze them to determine what kind of results are possible from them. How should we measure student understanding given the complexity and unique features of different formal education systems?  One way is to create a "scorecard" based on what the learning sciences claim will lead to high levels of understanding. Recall, in particular, the five principles from learning science about the conditions required to develop authentic understanding:

1. Authentic understanding depends on hierarchically organized knowledge.
2. Authentic understanding is grounded in direct experience.
3. Authentic understanding is stabilized by practice (generally at every level within the hierarchy).
4. Authentic understanding requires formative feedback.
5. Authentic understanding is context-sensitive.

The table below is an "understanding scorecard" that summarizes the principles and offers some examples of how to use each principle as a rating criterion.  We invite you to try out the scorecard for the first lesson Khan created to introduce the notions and elements of arithmetic.

Watching the video takes about 8 minutes.  Afterwards, see if rating the video as Low, Medium, or High on each of the five principles helps you summarize your reflections on the overall level of understanding we might expect from students using the video as an instructional tool.  Of course, the more videos you watch, the easier it will be to generate a summary evaluation of the arithmetic curriculum.

Evaluation criterion	Examples of arithmetic activities supporting “high” understanding	Your_Rating of Khan_Academy Arithmetic
Learning is grounded in experience	Hands-on learning experiences using [familiar objects like] chips, dice, or paper clips to associate physical objects to ordering, counting and symbols used to represent numbers.
Knowledge is hierarchically developed from the student’s point of view.	Concepts learned in a hierarchical way: Understandings begin as actions (as above), which precede and eventually support understandings that are representations of actions (writing, speaking or drawing), which in time support understandings that coordinate numerous representations to form abstractions (like justice or calculus).  If you want to know more about hierarchies of understanding see this article (pages 3-4).
Provides scaffolded practice (preferably at every level within the hierarchy)	The curriculum covers fewer concepts, so students can spend significant time practicing with physical objects (chips, dice, etc.) then with drawing pictures, then with symbols.  The teacher helps them as necessary (provides scaffolding) during this practice at every level of the hierarchy.
Provides formative feedback	As students practice with physical objects (chips, dice, etc.) then with drawing pictures, then with symbols, their level of understanding is made visible to themselves and the teacher, which creates opportunities for providing very specific corrective feedback when a student gets stuck or misunderstands (this is formative feedback).
Develops connections between abstract principles and real-world contexts	The abstract principles are numbers, operations, and the other symbolic formalisms of math.  Students spend a lot of time developing connections between these abstract principles and real world scenarios that they are used to model.

How useful does the scorecard seem to you?

As you complete the scorecard, it might also help to consider some of the following questions (from a first grader’s point of view):

• Do you need to know what an avocado is to make sense of the instruction?
• How important is it that the avocado looks like an avocado on video?
• How comfortable does the child already need to be with the idea that the number “2” has a special relationship with the two avocados that Khan draws?
• What do the symbols “+” or “=” mean as used in Khan’s lesson?

If you’d like, try using the scorecard to assess the highest level of understanding that Khan Academy supports, and then compare your response to others' (using "Show results").

Now we invite you to use the scorecard to evaluate a curriculum you're familiar with - at your child's school, the entire school, a program you recently went through, etc. and then compare your response with others'.

What Do You Think?

• Does the scorecard help you think more clearly about what you and others mean by “understanding”?
• How did you rate Khan Academy on the scorecard?  Were you surprised by others’ responses?
• How did you rate your own schools on the scorecard?  Were you surprised by others’ responses?

Khan Academy: The Illusion of Understanding (Part 1)

Guest blog by Dr. Marc Schwartz
Professor of Education at the University of Texas at Arlington
Director of the Southwest Center for Mind, Brain, and Education
This post is based on an article by the same name published in the Journal of Asynchronous Learning Networks.

The Illusion of Understanding
For the past three decades I’ve been working to dispel myself of an illusion that’s hard to recognize and even harder to overcome. I call it the “Illusion of Understanding.”  It’s the false belief that we understand something but then we discover we actually don’t.  The example problem below will help clarify what I mean.  Consider the following…

Iceberg Challenge

The glass of water in this picture is filled to the brim.  One more drop, and water would spill over the edge.  When examining the ice you note that the cubes rise just above the surface of the water (like glaciers in the ocean), but do not extend to the bottom of the glass.  Now here’s the challenge: Imagine patiently waiting on a hot summer day until all the ice melts.  What will happen to the water level?  Does it rise and over-flow the glass, remain constant throughout the melting process, or go down?

What do you think will happen to the water level when all the ice has melted?

Think about what’s going on for you as you wrestle with this challenge.  Do you feel like you know the right answer? How confident are you in your response? Are you, like most people who face this challenge, surprised to find that you aren’t sure of the answer, while also feeling conflicted because you think you should know it?  If you answered “Yes” to this last question, then you just experienced the Illusion of Understanding first-hand.

This is a challenging problem for most people – physics students and adults alike. Yet the problem is based on a principle called Archimedes Principle that most of us encountered at some point in a physical science class.

As challenging as the problem is for students, consider how much more daunting it is for a science teacher who wants to help students understand the principle so well that even years later they can confidently use that understanding to solve problems like this one.  I know how daunting the challenge is, because many years ago I was that science teacher.

Here’s the dilemma. As a teacher, I can assure you that it’s very, very difficult to help students develop what I call “authentic understanding” – the kind of understanding that would enable them to answer the iceberg challenge correctly and explain why their answer is correct.  It’s a great deal easier (although not a conscious goal) for a teacher to leave students with the illusion of understanding – the belief that they understand the relevant principle even though they can’t answer questions based on the principle.  I have given the iceberg challenge to hundreds of intelligent, educated adults over many years. Based on their performance I’d say that – despite the best efforts of many capable, dedicated science teachers – authentic understanding in this subject area is relatively rare while the illusion of understanding is quite common.  However if you ask people in the grip of the Illusion whether they understood their science teacher’s lesson on Archimedes’ Principle, many would say “yes” without hesitation.

MOOCs to the rescue?
Let’s carry our self-experiment one step further to see how deep the Illusion goes in this case.  Perhaps Kahn Academy can help you solve the iceberg challenge (assuming schooling has not). Khan’s curriculum on Fluids, Part 5 and Part 6, constitute a formal presentation that, in principle, should allow you to solve the problem as posed above. I invite you to watch those two videos now and try to answer the Iceberg Challenge again.

(Go ahead and watch the videos now.  I'll wait…)

How did you do?  Were you able to resolve the iceberg challenge? Do you feel more or less confident in your answer now?  Khan claims that the ability to control the pacing of the video and the opportunity to re-watch the session will help. You may want to test those assumptions.

How to Develop Authentic Understanding
I have found that Khan – like the many others who use similar instructional strategies both online and off – are overlooking over a hundred years’ worth of discoveries in the learning sciences. Below, I list five major discoveries that define requirements for achieving authentic understanding (see the companion article published in this month’s Journal of Asynchronous Learning Networks for additional detail):

1. Authentic understanding depends on hierarchically organized knowledge.
2. Authentic understanding is grounded in direct experience.
3. Authentic understanding is stabilized by practice (generally at every level within the hierarchy).
4. Authentic understanding requires formative feedback.
5. Authentic understanding is context-sensitive.

When these insights are ignored, as they are most of the time in online instruction, educators and students risk reinforcing and perpetuating the Illusion of Understanding, which I have observed in many classrooms and in many countries.  Typically the illusion unfolds in dramatic fashion when teachers ask students to explain their answers, and the students suddenly realize they can’t. Students find themselves speechless or stuck in a rambling explanation that doesn’t even make sense to them.  I have even observed students give a correct explanation and then admit they didn’t understand what they just said.  Now, as the teaching and learning enterprise unfolds on a world stage through a variety of online platforms, we face the risk that the Illusion will be even more widespread and difficult to dispel than ever.

Assuming that your struggle with the above iceberg challenge is no different than almost everyone who attempts the challenge, which of the five observations seem relevant? Did you feel that you lacked the perquisite experiences to reach an answer? Did your past experiences feel relevant yet unconnected or unreliable? Did you feel like you needed to practice some kind of mental exercise but did not know which or how?  These questions all underlie the complicated teaching and learning environment that lead to authentic understanding.  I don’t claim that the process is easy, but the investment is necessary. The challenge here should also underscore another illusion- that is, the illusion that achieving expert status in one discipline - as a hedge fund manager, for example - automatically transfers to another discipline, such as teaching. Teachers, like hedge fund managers, spend decades to become competent at their craft.

If you do watch the videos, which of the five observations seem to be relevant to your experience of understanding?  You will note that Khan does use a similar challenge in parts five and six of his video series, but the context is different.  Does that matter? In Part II we explore in further detail the Illusion of Understanding in the area of math and explore what choices may be available to Khan and all educators, especially those who work online, to better support authentic understanding.

(Continued in Part II which is available here.)