Khan Academy: How Does It Measure Up? (Part 2 of 2)

This article is the second in a series. In the first article, Dr. Schwartz distinguished authentic understanding from the Illusion of Understanding and introduced five principles from learning science that support the development of authentic understanding. For further reading, check out this article published in the Journal of Asynchronous Networks.

Note: The original published version of this post was longer.  We shortened it in response to reader feedback.  If you'd like to read the original version of the post, please contact the blog owner.

Dr. Marc Schwartz 
Professor of Education at the University of Texas at Arlington and Director of the Southwest Center for Mind, Brain, and Education

Dr. Michael Connell
Ed Tech Designer & Visiting Researcher at the University of Texas’ Southwest Center for Mind, Brain, and Education

Introduction

In Part 1 of this article I posed a challenge you may still be considering. If you remember the Iceberg Challenge, your goal was to decide what would happen to the water level after all the ice melted.  For many years, what nearly all my students found particularly irritating about this challenge (and me) is that I stopped providing the answer. You might be feeling that irritation too.  

If you read Part 1 and tried the challenge, are you feeling this irritation?

My goal – then or now – was not to be irritating. My goal is to use our collective experience of the iceberg challenge to clarify what we mean when we use the word understanding, so that we’re all talking about the same phenomenon in the same way with the same expectations. 

In Part II we introduce an understanding scorecard to help expose the Illusion of Understanding and in turn define what understanding means in the area of math, and finally consider what choices may be available to Khan and all educators, especially those who work online, to better support authentic understanding.

What do we mean by understanding?

At one extreme, an understanding might mean that we know something (anything) about a subject, so that we can participate in a cocktail conversation. For example, imagine a person said to you, “I think it makes a difference to coast lines if all the polar ice floating on the oceans melts, but I’m not sure how…” Would you say that person understands Archimedes Principle? 

Alternatively, would you say a person understands Archimedes Principle if they can provide a definition or use a mathematical formula to solve for a missing variable? 

At the other extreme, would you say a person understands Archimedes Principle who can:

• Recognize deeper connections between situations that seem unrelated on the surface - such as what happens to ice melting in a glass and what happens to a balloon full of oxygen released on Mars (whose atmosphere is predominantly carbon dioxide),
• Solve a variety of novel challenges like the Iceberg Challenge,
• Explain their reasoning and articulate why they believe their answers are correct across different contexts, and
• Recognize how a new concept or formula relates to what they have learned previously, so they can start using it quickly? 

These three points of view (let’s call them “low,” “medium,” and “high” understanding, respectively) map out positions along a continuum that begin to portray understanding in a richer and more complex way. We may all discover that in the past we have been holding different assumptions when using words like “understanding” (or “learning,” for that matter).

Do these three points of view frame a continuum that feels useful to you?

Using this continuum as a shared point of reference, we can ask a couple of distinct but related questions:

• What outcomes are possible? What is the highest level of understanding that students can theoretically achieve in a given subject area on a large scale in a particular formal education system, given the available resources in that system?
• What outcomes are expected? What level of student understanding should we hold the formal education system accountable for in practice?

The two questions above may seem similar but they could hardly be more different. The first question is a question of fact – there is an objective answer independent of what we believe or desire (although it might be difficult to discover that answer – more on that later). While different people might have different beliefs about the answer to the first question, at least one of them is guaranteed to be wrong. The second question is not a matter of fact – it is open-ended and requires a community decision. Different people and communities will certainly have different views about the second question and – as long as they respect the objective limits on what is possible – none of them can be considered wrong because there is no objectively right answer.

Even though the two questions are distinct, they are related. The first question (what is theoretically possible) puts a hard limit on reasonable answers to the second question (what the community demands of its educational system). Two common mistakes that people make when reasoning about education are: 

• They assume a low level of understanding is the best that can be achieved at scale in an education system, and – without checking that assumption – they decide to set a low bar for student understanding based on it.
• Conversely, they ignore the ceiling on what is theoretically possible and make impossible demands of educational institutions. 

It is not our aim here to argue in favor of or against any particular purpose of education. What is important right now is to know what we mean when we say we want students to understand _______ (and you fill in the blank), and to be clear about which question we are discussing at any given time (what’s possible vs. what's expected).

How do we determine what level of understanding is possible?

Formal education systems are so complex that it is difficult to analyze them to determine what kind of results are possible from them. How should we measure student understanding given the complexity and unique features of different formal education systems?  One way is to create a "scorecard" based on what the learning sciences claim will lead to high levels of understanding. Recall, in particular, the five principles from learning science about the conditions required to develop authentic understanding:

1. Authentic understanding depends on hierarchically organized knowledge.
2. Authentic understanding is grounded in direct experience.
3. Authentic understanding is stabilized by practice (generally at every level within the hierarchy).
4. Authentic understanding requires formative feedback.
5. Authentic understanding is context-sensitive.

The table below is an "understanding scorecard" that summarizes the principles and offers some examples of how to use each principle as a rating criterion.  We invite you to try out the scorecard for the first lesson Khan created to introduce the notions and elements of arithmetic.

Watching the video takes about 8 minutes.  Afterwards, see if rating the video as Low, Medium, or High on each of the five principles helps you summarize your reflections on the overall level of understanding we might expect from students using the video as an instructional tool.  Of course, the more videos you watch, the easier it will be to generate a summary evaluation of the arithmetic curriculum.

Evaluation criterion	Examples of arithmetic activities supporting “high” understanding	Your_Rating of Khan_Academy Arithmetic
Learning is grounded in experience	Hands-on learning experiences using [familiar objects like] chips, dice, or paper clips to associate physical objects to ordering, counting and symbols used to represent numbers.
Knowledge is hierarchically developed from the student’s point of view.	Concepts learned in a hierarchical way: Understandings begin as actions (as above), which precede and eventually support understandings that are representations of actions (writing, speaking or drawing), which in time support understandings that coordinate numerous representations to form abstractions (like justice or calculus).  If you want to know more about hierarchies of understanding see this article (pages 3-4).
Provides scaffolded practice (preferably at every level within the hierarchy)	The curriculum covers fewer concepts, so students can spend significant time practicing with physical objects (chips, dice, etc.) then with drawing pictures, then with symbols.  The teacher helps them as necessary (provides scaffolding) during this practice at every level of the hierarchy.
Provides formative feedback	As students practice with physical objects (chips, dice, etc.) then with drawing pictures, then with symbols, their level of understanding is made visible to themselves and the teacher, which creates opportunities for providing very specific corrective feedback when a student gets stuck or misunderstands (this is formative feedback).
Develops connections between abstract principles and real-world contexts	The abstract principles are numbers, operations, and the other symbolic formalisms of math.  Students spend a lot of time developing connections between these abstract principles and real world scenarios that they are used to model.

How useful does the scorecard seem to you?

As you complete the scorecard, it might also help to consider some of the following questions (from a first grader’s point of view):

• Do you need to know what an avocado is to make sense of the instruction?
• How important is it that the avocado looks like an avocado on video?
• How comfortable does the child already need to be with the idea that the number “2” has a special relationship with the two avocados that Khan draws?
• What do the symbols “+” or “=” mean as used in Khan’s lesson?

If you’d like, try using the scorecard to assess the highest level of understanding that Khan Academy supports, and then compare your response to others' (using "Show results").

Now we invite you to use the scorecard to evaluate a curriculum you're familiar with - at your child's school, the entire school, a program you recently went through, etc. and then compare your response with others'.

What Do You Think?

• Does the scorecard help you think more clearly about what you and others mean by “understanding”?
• How did you rate Khan Academy on the scorecard?  Were you surprised by others’ responses?
• How did you rate your own schools on the scorecard?  Were you surprised by others’ responses?