Friday, March 21, 2014

Khan Academy: How Does It Measure Up? (Part 2 of 2)

This article is the second in a series. In the first article, Dr. Schwartz distinguished authentic understanding from the Illusion of Understanding and introduced five principles from learning science that support the development of authentic understanding. For further reading, check out this article published in the Journal of Asynchronous Networks.

Note: The original published version of this post was longer.  We shortened it in response to reader feedback.  If you'd like to read the original version of the post, please contact the blog owner.

Dr. Marc Schwartz 
Professor of Education at the University of Texas at Arlington and Director of the Southwest Center for Mind, Brain, and Education

Dr. Michael Connell
Ed Tech Designer & Visiting Researcher at the University of Texas’ Southwest Center for Mind, Brain, and Education


In Part 1 of this article I posed a challenge you may still be considering. If you remember the Iceberg Challenge, your goal was to decide what would happen to the water level after all the ice melted.  For many years, what nearly all my students found particularly irritating about this challenge (and me) is that I stopped providing the answer. You might be feeling that irritation too.  

If you read Part 1 and tried the challenge, are you feeling this irritation?
My goal – then or now – was not to be irritating. My goal is to use our collective experience of the iceberg challenge to clarify what we mean when we use the word understanding, so that we’re all talking about the same phenomenon in the same way with the same expectations. 

In Part II we introduce an understanding scorecard to help expose the Illusion of Understanding and in turn define what understanding means in the area of math, and finally consider what choices may be available to Khan and all educators, especially those who work online, to better support authentic understanding.

What do we mean by understanding?

At one extreme, an understanding might mean that we know something (anything) about a subject, so that we can participate in a cocktail conversation. For example, imagine a person said to you, “I think it makes a difference to coast lines if all the polar ice floating on the oceans melts, but I’m not sure how…” Would you say that person understands Archimedes Principle? 

Alternatively, would you say a person understands Archimedes Principle if they can provide a definition or use a mathematical formula to solve for a missing variable? 

At the other extreme, would you say a person understands Archimedes Principle who can:
  • Recognize deeper connections between situations that seem unrelated on the surface - such as what happens to ice melting in a glass and what happens to a balloon full of oxygen released on Mars (whose atmosphere is predominantly carbon dioxide),
  • Solve a variety of novel challenges like the Iceberg Challenge,
  • Explain their reasoning and articulate why they believe their answers are correct across different contexts, and
  • Recognize how a new concept or formula relates to what they have learned previously, so they can start using it quickly? 

These three points of view (let’s call them “low,” “medium,” and “high” understanding, respectively) map out positions along a continuum that begin to portray understanding in a richer and more complex way. We may all discover that in the past we have been holding different assumptions when using words like “understanding” (or “learning,” for that matter).

Do these three points of view frame a continuum that feels useful to you?

Using this continuum as a shared point of reference, we can ask a couple of distinct but related questions:
  • What outcomes are possible? What is the highest level of understanding that students can theoretically achieve in a given subject area on a large scale in a particular formal education system, given the available resources in that system?
  • What outcomes are expected? What level of student understanding should we hold the formal education system accountable for in practice?
The two questions above may seem similar but they could hardly be more different. The first question is a question of fact – there is an objective answer independent of what we believe or desire (although it might be difficult to discover that answer – more on that later). While different people might have different beliefs about the answer to the first question, at least one of them is guaranteed to be wrong. The second question is not a matter of fact – it is open-ended and requires a community decision. Different people and communities will certainly have different views about the second question and – as long as they respect the objective limits on what is possible – none of them can be considered wrong because there is no objectively right answer.

Even though the two questions are distinct, they are related. The first question (what is theoretically possible) puts a hard limit on reasonable answers to the second question (what the community demands of its educational system). Two common mistakes that people make when reasoning about education are: 
  • They assume a low level of understanding is the best that can be achieved at scale in an education system, and – without checking that assumption – they decide to set a low bar for student understanding based on it.
  • Conversely, they ignore the ceiling on what is theoretically possible and make impossible demands of educational institutions. 
It is not our aim here to argue in favor of or against any particular purpose of education. What is important right now is to know what we mean when we say we want students to understand _______ (and you fill in the blank), and to be clear about which question we are discussing at any given time (what’s possible vs. what's expected).

How do we determine what level of understanding is possible?

Formal education systems are so complex that it is difficult to analyze them to determine what kind of results are possible from them. How should we measure student understanding given the complexity and unique features of different formal education systems?  One way is to create a "scorecard" based on what the learning sciences claim will lead to high levels of understanding. Recall, in particular, the five principles from learning science about the conditions required to develop authentic understanding:
  1. Authentic understanding depends on hierarchically organized knowledge.
  2. Authentic understanding is grounded in direct experience.
  3. Authentic understanding is stabilized by practice (generally at every level within the hierarchy).
  4. Authentic understanding requires formative feedback.
  5. Authentic understanding is context-sensitive.
The table below is an "understanding scorecard" that summarizes the principles and offers some examples of how to use each principle as a rating criterion.  We invite you to try out the scorecard for the first lesson Khan created to introduce the notions and elements of arithmetic.

Watching the video takes about 8 minutes.  Afterwards, see if rating the video as LowMedium, or High on each of the five principles helps you summarize your reflections on the overall level of understanding we might expect from students using the video as an instructional tool.  Of course, the more videos you watch, the easier it will be to generate a summary evaluation of the arithmetic curriculum.

Evaluation criterion
Examples of arithmetic activities supporting “high” understanding
Your_Rating of Khan_Academy
Learning is grounded in experience
Hands-on learning experiences using [familiar objects like] chips, dice, or paper clips to associate physical objects to ordering, counting and symbols used to represent numbers.

Knowledge is hierarchically developed from the student’s point of view.
Concepts learned in a hierarchical way: Understandings begin as actions (as above), which precede and eventually support understandings that are representations of actions (writing, speaking or drawing), which in time support understandings that coordinate numerous representations to form abstractions (like justice or calculus).  If you want to know more about hierarchies of understanding see this article (pages 3-4). 

Provides scaffolded practice (preferably at every level within the hierarchy)
The curriculum covers fewer concepts, so students can spend significant time practicing with physical objects (chips, dice, etc.) then with drawing pictures, then with symbols.  The teacher helps them as necessary (provides scaffolding) during this practice at every level of the hierarchy.

Provides formative feedback
As students practice with physical objects (chips, dice, etc.) then with drawing pictures, then with symbols, their level of understanding is made visible to themselves and the teacher, which creates opportunities for providing very specific corrective feedback when a student gets stuck or misunderstands (this is formative feedback).

Develops connections between abstract principles and real-world contexts
The abstract principles are numbers, operations, and the other symbolic formalisms of math.  Students spend a lot of time developing connections between these abstract principles and real world scenarios that they are used to model.

How useful does the scorecard seem to you?

As you complete the scorecard, it might also help to consider some of the following questions (from a first grader’s point of view):
  • Do you need to know what an avocado is to make sense of the instruction?
  • How important is it that the avocado looks like an avocado on video?
  • How comfortable does the child already need to be with the idea that the number “2” has a special relationship with the two avocados that Khan draws?
  • What do the symbols “+” or “=” mean as used in Khan’s lesson?

If you’d like, try using the scorecard to assess the highest level of understanding that Khan Academy supports, and then compare your response to others' (using "Show results").

Now we invite you to use the scorecard to evaluate a curriculum you're familiar with - at your child's school, the entire school, a program you recently went through, etc. and then compare your response with others'.

What Do You Think?

  • Does the scorecard help you think more clearly about what you and others mean by “understanding”?
  • How did you rate Khan Academy on the scorecard?  Were you surprised by others’ responses?
  • How did you rate your own schools on the scorecard?  Were you surprised by others’ responses?


  1. I understand your frustration but unfortunately educators seem to feel that they can apply the same rules to everything to produce definitive answers. Thus to many educators the process of, for example, developing a math syllabus and integrating technology can be treated the same. Unfortunately it's apples and oranges and quite frankly most educators are out of their depth as far as the implications of changes in technology are concerned. Take for example the World Wide Web. It's been used in education for nearly 20 years but if you asked 100 educators to encapsulate in one word the essential benefit that makes it unique as an educational resource, how many could answer? One maybe? Yet less than 10 years ago everyone was convincing everyone else that Web 2.0 was going to revolutionize education. What happened? Time passed and the hyperbole of that period got put into perspective, is what happened. Unfortunately there is also a tendency in academia for people to jump on bandwagons in the hope of creating a reputation for themselves. Sometimes this results in a frenzy of debate 'full of sound and fury but signifying nothing' except extreme views. Your consternation about the validity of non-traditional teaching is absolutely spot on but entering into the debate from the standpoint that's it's all junk diminishes the persuasiveness of your argument somewhat. It would be naive to think that Khan isn't aware of the limitations imposed by the medium he has chosen but also, in answer to your question 'Otherwise, why would they all consult Khan Academy instead of any other potential contributor to education outcomes – such as expert teachers, or textbook providers, or subject matter experts, or learning scientists, or education researchers...', at least he's trying to do something not just talking about it.

    1. Hello, Mr. B.

      Thanks for your comments.

      This post isn't a critique of Khan Academy. We didn't say anything about the qualify of Khan Academy's offerings.

      We are offering a tool to the community (the scorecard) that we think might help support more productive conversations about education. We wanted to give people a chance to try the tool out for themselves and (hopefully) share their experience and thoughts with us.

    2. "..What happened?...hyperbole was put into perspective.."
      Perhaps reality set in, it's not about understanding of the content, it's understanding of the assessment.
      TIMMS and PISA has been criticised for mis representing what it measures. While the headlines describe poor achievement in maths and science (in Australia and USA) further de-briefing of our students show much of the issue is not their understanding of principles but their interpretation of the questions and their literacy.
      Another issue for Australianis our schools are randomly chosen and are not representative of best practice, merely average practice.

      As to Khan practices, they too recognise assessment is the key.
      The latest Kahn pronouncement s tell us that SAT and AP exam question solutions and how to write them will now be a focus in their math program.
      Not so much teach for understanding, but teach for getting a higher score.
      Singapore Math has had this as their objective from the beginning.
      Highly competitive systems generate schemes to deliver results.
      Japan has an enormous coaching industry geared towards University entry. In Australia, exam practice is a thriving industry, as are exam preparation 'camps'...

  2. I've watched Kahn videos out of curiosity of how high school/middle school might use these in “flipped” classrooms. I don’t think they are “evil” - a skilled teacher can use many various tools in various ways…see author's blog about using Minecraft.
    I teach 2nd, have taught PK-4 year olds & 1st grade. My immediate reaction was to scream “STOP! P-L-E-A-S-E for the love of learning…STOP!” What was covered in this < 8 min video is roughly 2/3 of 1st grade math objectives. Kahn Academy has a Common Core set of videos. I looked only at the demos. At first glance, it appeared they've taken care to break down the concepts in that early video. These were scary--no way would I EVER show these to my students. It's not appropriate for the authors of this blog to blast any product, but I can…One beef is the $$ spent developing these…$$ from foundations like the Gates Foundation. A branding has occurred with the name “Kahn” that receives, fairly or not, a positive response. Buyer beware, be diligent…even if products are FREE!
    But, this is not a comment about Kahn but about administrators: my early childhood colleagues and I find that many times administrators think that early childhood instruction is just like instruction for older students. They fail to take into account a child’s actual development. Little peeps aren’t just like 5th graders with shorter legs! Problem 1

    “We” think that we can design a scope of learning objectives and place it within a contrived timeline, based on a contrived school calendar, without taking into account previous or current understanding. This loops back to Problem 1 and development. One district I know allotted 3 days to teach place value in 1st grade. Think about that. This same district disallowed adding an instructional tool for number sense 15 minutes a day during the last grading period of kindergarten because "it was taking away instructional time from that grading period’s learning objectives…which were covered in the 1st grading period.” How does that absurd constraint relate to the score card? Problem 2

    A fairly “good” curriculum, one that upon first glance appears it would receive high scores with the authors’ score card? Many times districts give teachers a curriculum, often without the professional development or time to digest the curricula in a way that allows them to create lessons/activities that support those curricula. Problem 3

    3.a Learning occurs on the teachers’ end before it can be translated to the students! There’s plenty of research showing many teachers don’t feel comfortable with their own level of content knowledge in particular subjects; which might suggest the exact reason some teachers are supportive of Kahn type learning…it’s fast and easier than trying to figure out the long, in-depth, complicated background info provided in text books/curricula. Embedded within this problem is the lack of planning time available and lack of pre-service (undergraduate) instruction. All complex problems even when a district adopts a “good” curriculum.

    What do teachers do…they collaborate on their own, and if the numbers suggest anything, this collaboration occurs on-line via websites like Pinterest, Teachers Pay Teachers and teachers’ individual blogs. I have done this, and will continue to do this, because several (I am NOT suggesting ALL) of these teachers are true experts in their fields and have synthesized the understanding needed by BOTH the teacher and the student. It is a hybrid community of practice.

    Problem 4: Assessments. Way too in-depth to cover in this blog, but the scorecard gets to the meat of this: what are we teaching and assessing; why, how, and what do we do with that data?! I like the score card as a baseline. It can be used by individual teachers regardless of adopted curriculum; it can measure one’s own teaching practice, potential effectiveness of content in a curriculum, and I think it can be adapted to measure student outcomes.