Thursday, February 20, 2014

Khan Academy: The Illusion of Understanding (Part 1)


Guest blog by Dr. Marc Schwartz
Professor of Education at the University of Texas at Arlington
Director of the Southwest Center for Mind, Brain, and Education
This post is based on an article by the same name published in the Journal of Asynchronous Learning Networks.

The Illusion of Understanding
For the past three decades I’ve been working to dispel myself of an illusion that’s hard to recognize and even harder to overcome. I call it the “Illusion of Understanding.”  It’s the false belief that we understand something but then we discover we actually don’t.  The example problem below will help clarify what I mean.  Consider the following…

Iceberg Challenge
Glass of ice water with ice cubes floating in it
The glass of water in this picture is filled to the brim.  One more drop, and water would spill over the edge.  When examining the ice you note that the cubes rise just above the surface of the water (like glaciers in the ocean), but do not extend to the bottom of the glass.  Now here’s the challenge: Imagine patiently waiting on a hot summer day until all the ice melts.  What will happen to the water level?  Does it rise and over-flow the glass, remain constant throughout the melting process, or go down?

What do you think will happen to the water level when all the ice has melted?


Think about what’s going on for you as you wrestle with this challenge.  Do you feel like you know the right answer? How confident are you in your response? Are you, like most people who face this challenge, surprised to find that you aren’t sure of the answer, while also feeling conflicted because you think you should know it?  If you answered “Yes” to this last question, then you just experienced the Illusion of Understanding first-hand.

This is a challenging problem for most people – physics students and adults alike. Yet the problem is based on a principle called Archimedes Principle that most of us encountered at some point in a physical science class.

As challenging as the problem is for students, consider how much more daunting it is for a science teacher who wants to help students understand the principle so well that even years later they can confidently use that understanding to solve problems like this one.  I know how daunting the challenge is, because many years ago I was that science teacher.

Here’s the dilemma. As a teacher, I can assure you that it’s very, very difficult to help students develop what I call “authentic understanding” – the kind of understanding that would enable them to answer the iceberg challenge correctly and explain why their answer is correct.  It’s a great deal easier (although not a conscious goal) for a teacher to leave students with the illusion of understanding – the belief that they understand the relevant principle even though they can’t answer questions based on the principle.  I have given the iceberg challenge to hundreds of intelligent, educated adults over many years. Based on their performance I’d say that – despite the best efforts of many capable, dedicated science teachers – authentic understanding in this subject area is relatively rare while the illusion of understanding is quite common.  However if you ask people in the grip of the Illusion whether they understood their science teacher’s lesson on Archimedes’ Principle, many would say “yes” without hesitation.

MOOCs to the rescue?
Let’s carry our self-experiment one step further to see how deep the Illusion goes in this case.  Perhaps Kahn Academy can help you solve the iceberg challenge (assuming schooling has not). Khan’s curriculum on Fluids, Part 5 and Part 6, constitute a formal presentation that, in principle, should allow you to solve the problem as posed above. I invite you to watch those two videos now and try to answer the Iceberg Challenge again.

(Go ahead and watch the videos now.  I'll wait…)

How did you do?  Were you able to resolve the iceberg challenge? Do you feel more or less confident in your answer now?  Khan claims that the ability to control the pacing of the video and the opportunity to re-watch the session will help. You may want to test those assumptions.

How to Develop Authentic Understanding
I have found that Khan – like the many others who use similar instructional strategies both online and off – are overlooking over a hundred years’ worth of discoveries in the learning sciences. Below, I list five major discoveries that define requirements for achieving authentic understanding (see the companion article published in this month’s Journal of Asynchronous Learning Networks for additional detail):
  1. Authentic understanding depends on hierarchically organized knowledge.
  2. Authentic understanding is grounded in direct experience.
  3. Authentic understanding is stabilized by practice (generally at every level within the hierarchy).
  4. Authentic understanding requires formative feedback.
  5. Authentic understanding is context-sensitive.
When these insights are ignored, as they are most of the time in online instruction, educators and students risk reinforcing and perpetuating the Illusion of Understanding, which I have observed in many classrooms and in many countries.  Typically the illusion unfolds in dramatic fashion when teachers ask students to explain their answers, and the students suddenly realize they can’t. Students find themselves speechless or stuck in a rambling explanation that doesn’t even make sense to them.  I have even observed students give a correct explanation and then admit they didn’t understand what they just said.  Now, as the teaching and learning enterprise unfolds on a world stage through a variety of online platforms, we face the risk that the Illusion will be even more widespread and difficult to dispel than ever.

Assuming that your struggle with the above iceberg challenge is no different than almost everyone who attempts the challenge, which of the five observations seem relevant? Did you feel that you lacked the perquisite experiences to reach an answer? Did your past experiences feel relevant yet unconnected or unreliable? Did you feel like you needed to practice some kind of mental exercise but did not know which or how?  These questions all underlie the complicated teaching and learning environment that lead to authentic understanding.  I don’t claim that the process is easy, but the investment is necessary. The challenge here should also underscore another illusion- that is, the illusion that achieving expert status in one discipline - as a hedge fund manager, for example - automatically transfers to another discipline, such as teaching. Teachers, like hedge fund managers, spend decades to become competent at their craft.

If you do watch the videos, which of the five observations seem to be relevant to your experience of understanding?  You will note that Khan does use a similar challenge in parts five and six of his video series, but the context is different.  Does that matter? In Part II we explore in further detail the Illusion of Understanding in the area of math and explore what choices may be available to Khan and all educators, especially those who work online, to better support authentic understanding.

(Continued in Part II which is available here.)

24 comments:

  1. Replies
    1. Thanks, Proofrock.
      (I appreciate the homage to T.S. Eliot, btw).

      We're working on it as we speak!

      Delete
  2. Thank you for writing this - very well done!

    I would recommend this video, from Derek Muller (Veritasium) on a related topic: "Khan Academy and the Effectiveness of Science Videos"

    http://www.youtube.com/watch?v=eVtCO84MDj8&list=PL772556F1EFC4D01C&index=26

    Very germane to this discussion.

    ReplyDelete
    Replies
    1. Thanks for sharing the link, Mike. I will definitely check out this video.

      Delete
    2. I watched the video and you're right - it's *very* relevant to this discussion. Thanks much for pointing us to it!

      Delete
  3. Have you looked at Hubbard's research on the subject of learning, specifically what he called the "Three barriers to study?"
    I've been using that stuff all my life and it is gold!
    http://www.appliedscholastics.org/study-tech/a-breakthrough-in-learning.html

    ReplyDelete
    Replies
    1. This is new to me, Florian. Thanks for sharing - I will take a look.

      Delete
  4. I could be wrong, but in my opinion you fall for an illusion yourself. There is a difference between understanding and skill. One is a variable and the other is a constant. When I have to solve a short task, why should I reduce the liquidity of my brain and imprint this solution as the median of all similar tasks although I do not know the range?
    Same goes for your example about authentic understanding when students cannot give an answer although they should know, have they been taught the range of tasks? Have they been taught the range of solutions? Was there a chance to create a pattern, an emergent knowledge?
    And how much does this affect my life when I have to pay a rent or had a big argument with my love? Emotions easily override and overwrite any common sense. That's why so many child prodigies fail once they reach puberty. Conclusion, if you want authentic understanding of a direct connection between task and solution, then why is science and education so emotionless? And why is this webpage clinically black and white?

    ReplyDelete
    Replies
    1. Hi, Harry. Thanks for joining the conversation.

      > Same goes for your example about authentic understanding when students cannot give an answer although they should know

      Just to be clear: the point made in the blog is not that "students cannot give an answer although they should know [the answer]." The situation is that students *believe* they know the relevant concept but are themselves surprised and dismayed to discover that they *don't* actually know it.

      Delete
  5. Powerful,argument Marc,you have dared where many haven't. I find a lot of sense in your concept of authentic understanding. Your concept is supported by the schema theory: schema activation, where the topic has to be introduced via various strategies, schema construction where new concepts are derived principles are connected, and lastly on refinement, where we expect the student or learner to demonstrate transfer of knowledge is solving a problem. Okey 9 steps of learning cover the learning process along similar lines.

    ReplyDelete
  6. I see at least three topics to discuss in this blog/comments:
    1) Effectiveness of video instruction: Are they effective? In what context? What evidence exits that this model is more (or less) effective than another? Why would an instructor use this tool over another? Why would a student use this tool over another?
    2) Illusion of Understanding: Has this happened to anyone posting (or have you observed this?) How do our current methods of assessment add to this illusion? Do they add to this illusion? What tools exist that allow teachers to assess understanding that is authentic?
    4) Let’s assume that the illusion of understanding permeates our current education system PK-16 (and, btw, I believe it does) in what ways can we provide instruction to instructors to help them recognize this and, more importantly, reverse or remediate instructional models that add to this illusion?
    It is well and good (and necessary!) to discuss these issues, but what we need is action based on discoveries. That begins with first acknowledging this in ourselves; and being mindful of when this happens! Daniel Kahneman provides a stellar analysis of how we fall prey to the illusion of understanding in Thinking: Fast and Slow. This issue isn’t just one related to the world of education. The illusion of understanding affects our political views, our personal choices, our lives…
    What are you doing TODAY to begin to recognize and change your own illusions?

    ReplyDelete
    Replies
    1. These are great questions, Rene. In Part 2 we'll start to wrestle with some of them and we would love to hear your thoughts on that.

      Delete
  7. Oops...yes, I can count...just did not check for spelling/typos...
    Looking forward to further discussion and follow-on article!

    ReplyDelete
  8. What's the answer? You're killing me here. I'm guessing that the buoyancy of ice is inversely proportional to the density compared to water, which is lower, so as it melts, the water will stay at the same height.

    ReplyDelete
    Replies
    1. Hi, Dave.
      I know how you feel. :)

      We don't want to irritate anyone by withholding the answer. But we also believe that giving you the answer would actually do you and anyone reading this a disservice. If we gave you the answer, we'd be robbing you of the opportunity for authentic understanding. That's not something one person can give to another directly like that.

      Delete
  9. I found this an immensely interesting article which really made me question my own teaching methods and how I might redesign materials to encourage the understanding that is preferable. I am very much looking forward to Part II and the further challenges it will present me with as a Lecturer. Thank you .

    ReplyDelete
    Replies
    1. Thanks for the feedback, Angela. This is really wonderful to hear.

      Delete
  10. All good discussions and worth exploring but to me there are two illusions that contradict a personal philosophy about education and learning. the Khan approach makes assumptions about student access to their resources and ignores some basic social justice principals associated with many school districts. many students still cannon access the internet at home..its a cost issue and secondly many have to act as carers as both parents are at work. A truly authentic flip should be totally during school time for it to be equitable. I see many examples in the US that are contrived and do not address low socio economic schools needs. The second illusion is that Khan can substitute for real hands on learning that focuses upon the lifeworlds of a particular cohort of students.

    Enough for now...Your thoughts!

    ReplyDelete
  11. Online learning?

    I don't think that "watching videos" is the same as online learning!

    We all agree that learning needs context and many tools of online learning can carry a fair part of the context. Think of LMS, forums, or even portfolios.

    I like the impression "false understanding"... but that is something that also happens if you learn math recipes by hand. The criteria listed here make sure, for example, that there are varied exercises to be performed as part of a normal math or physics learning. Makes sense, is commonly applied.

    ReplyDelete
    Replies
    1. Hi, polx.

      > I don't think that "watching videos" is the same as online learning!

      You raise an interesting question. Currently, people use the term "online learning" to refer to a wide range of activities. The only thing they seem to have in common is that they require internet access. Do you have thoughts on how we might make a more precise definition of the term?

      Delete
  12. Michael,

    I wonder if this should be much constrained.
    Maybe... A learning process with the usage of a computer accessing the internet?

    paul

    ReplyDelete
  13. There is a trick in this question. Water behaves differently when it's ice, so this was confusing. I still don't know if this is relevant.

    Apologies, but I have read the "authentic understanding" concept before, and I do not see how it can be operationally defined. "Accurate understanding" might be more descriptive. The opposite is the "illusion of understanding (Robert Bjork 1996 or so) or "illusion of knowledge", Glenberg, 1982.)

    ReplyDelete
    Replies
    1. Hi, haa.

      There's no trick in the question. It's intended to be a straightforward question about physics.

      Thanks for these references to Bjork and Glenberg. I wasn't aware of their work but will look it up. I did find this blog post that discusses "Illusions of Understanding and Knowledge" after searching for Bjork based on your suggestion:
      http://www.edge.org/response-detail/23731

      It looks interesting and I will follow up more, so thanks for sharing those.

      As for operationally defining authentic understanding, Part 2 (which should be available later today) will address that very question. I hope you will let us know what you think after you read it!

      Delete
  14. Part 2 of the blog post is up now:
    http://theeducationscientist.blogspot.com/2014/03/khan-academy-how-does-it-measure-up.html

    ReplyDelete