It was an article in the New York Times titled “How Tests Make Us Smarter” that caused me to start thinking about developing a quizzing platform for ASimpleModel.com. What caught my attention was the following quote:
When [we] took our research out of the lab and into a … middle school class, we found that students earned an average grade of A- on material that had been presented in class once and subsequently quizzed three times, compared with a C+ on material that had been presented in the same way and reviewed three times but not quizzed. The benefit of quizzing remained in a follow-up test eight months later. [1]
Up to that point ASimpleModel had been designed to make the material easily accessible to permit revisiting content every time something was forgotten, thereby further committing knowledge to memory. But the quote suggested that frequent exposure to the same material was not the most effective learning process. In fact, the author took it one step further:
Surprisingly, researchers have also found that the most common study strategies — like underlining, highlighting and rereading — create illusions of mastery but are largely wasted effort, because they do not involve practice in accessing or applying what the students know. [2]
For an entertaining example, and if you remain convinced that frequent exposure is sufficient to commit knowledge to memory, Google “penny memory test.” As an aside, I failed the test that results from this search. The author continues:
When students are tested, they are required to retrieve knowledge from memory. Much educational activity, such as lectures and textbook readings, is aimed at helping students acquire and store knowledge. Various kinds of testing, though, when used appropriately, encourage students to practice the valuable skill of retrieving and using knowledge. The fact of improved retention after a quiz — called the testing effect or the retrieval practice effect — makes the learning stronger and embeds it more securely in memory. [3]
The author, Henry L. Roediger III, had previously co-authored Make it Stick: The Science of Successful Learning with Peter C. Brown and Mark A. McDaniel. I purchased the book after reading the article, and started down a path of learning about learning. I cannot tell you how much I wish I had this knowledge when I was in college.
There is an abundance of material, and much that should be shared. But in this post I have focused on the three categories that I believe have the greatest impact.
- Vocabulary: Learn the language.
- Deliberate Practice: Practice does not make perfect. Deliberate practice makes perfect.
- Testing & Feedback: It’s not a grade, it’s a learning process.
To provide additional context for the suggestions made, I also thought it would be helpful to apply the learning process to a new skill: financial modeling (of course). Regardless, I think the content that follows is relevant to learning almost anything new. Let’s get started.
VOCABULARY:
Some disciplines require learning a new language before much real progress can be made. ASimpleModel attempts to strip out all unnecessary accounting and financial vocabulary to teach financial modeling, but it means the new words introduced are critical. Commit yourself to learning this new language. The fact of the matter is that this knowledge base is essential to learning the financial modeling skill set. To make my case consider the following study:
“This paradox – it takes knowledge to gain knowledge – is captured in a study in which researchers wrote up a detailed description of a half inning of baseball and gave it to a group of baseball fanatics … and a group of less avid fans to read. The baseball fanatics structured their recollections around important game-related events, like runners advancing and runs scoring. They were able to reconstruct the half inning in sharp detail. One almost got the impression they were reading off an internal scorecard. The less avid fans remembered fewer important facts about the game and were more likely to recount superficial details like the weather. … Without a conceptual framework in which to embed what they were learning, they were effectively amnesics.” [4]
The greater your facility with the vocabulary required to learn a new subject, the faster you will absorb new related concepts. It is not just a matter of learning definitions, but of encountering the vocabulary in different context. The more associations you can make, the more likely you are to recall the meaning of the word. A good example is known as the Baker / baker paradox, which resonates with people that express difficulty remembering names. In this example two individuals are shown the same photograph of a face. One is told that the name of the person pictured is Baker, and the other is told that the person pictured is a baker. The individual that is told a profession instead of a name has a much higher chance of recalling the word at a later date when they are shown the same picture. So what is taking place?
“When you hear that the man in the photo is a baker, that fact gets embedded in a whole network of ideas about what it means to be a baker: He cooks bread, he wears a big white hat, he smells good when he comes home from work. The name Baker, on the other hand, is tethered only to a memory of the person’s face.” [5]
The purpose of keeping ASM material easily accessible at all times is to encourage accessing it as you encounter unique situations in your everyday life or career that require a reference. This exposes the material in different context. Each time you do this you create an additional “hook” to retrieve that information.
Make it Stick devotes a chapter to anecdotal evidence for the power of learning how to learn. One description in particular caught my attention, that of Michael Young, a medical student who had bypassed the typically required premed coursework and found himself floundering to keep up with classmates that had backgrounds in biochemistry and pharmacology. After barely achieving a 65 on his first exam he started studying learning and even reached out to the authors of Make it Stick. Per the author’s account, he describes it as follows:
“I was big into reading, but that’s all I knew how to do for studying. I would just read the material and I wouldn’t know what else to do with it. So if I read it and it didn’t stick in my memory, then I didn’t know what to do about that. What I learned from reading the research [on learning] is that you have to do something beyond just passively taking in the information.
Of course the big thing is to figure out a way to retrieve the information from memory, because that’s what you’re going to be asked to do on the test. If you can’t do it while you’re studying, then you’re not going to be able to do it on the test.”
He became more mindful of that when he studied. “I would stop. ‘Okay, what did I just read? What is this about?’ I’d have to think about it. ‘Well, I believe it happens this way: The enzyme does this, and then it does that.’ And then I’d have to go back and check if I was way off base or on the right track.” [6]
The text elaborates on his process, but the conclusion is telling. By the time he started his second year Michael Young had joined the high performers in his class, and he remained there through graduation. What I have omitted from the quote is that Young describes this process as very challenging. It did not feel natural to learn this way. This will become a common theme of effective learning.
DELIBERATE PRACTICE:
Anders Ericsson is considered by most to be the world’s foremost researcher on gaining expertise, and he will tell you that practice does not make perfect. Deliberate practice makes perfect.
Ericsson is mentioned in almost every text that I have read on the subject, so I finally purchased his: The Cambridge Handbook of Expertise and Expert Performance (the “Handbook”). It’s a 900 page phone-book of dense academic writing, but it is pretty awesome. It would lead one to believe that for every skill set on this planet, someone has studied the most efficient means of acquiring it. The Handbook addresses playing guitar, mathematical expertise, chess, software design, throwing darts, writing, etc.
To understand what deliberate practice means, it helps to understand how people acquire skill sets to begin with. The text outlines three steps (as you read through these think of your experience learning to type):
- In phase one, when an individual is first introduced to a skilled activity, the objective is to understand the basic requirements and achieve a basic level of proficiency. Most of the focus is on generating the desired result while minimizing mistakes.
- In phase two, after about 50 hours of practice, an acceptable level of performance is achieved and errors become far less frequent.
- In phase three automation sets in. The individual can now execute smoothly and with minimal effort. It is this automated approach to execution that makes advancing the skill set more difficult.
According to the Handbook, once phase three has been achieved an individual’s skill set will plateau indefinitely or even regress if the individual does not engage in deliberate practice. Part of the problem is that you forget why things work the way they do, and the steps required to learn the process initially. For example, could you articulate how you tie your shoelaces?
What this means is that there is little correlation between experience and skill set, which is a wild thought. In particular when many professionals are hired based on tenure. Ericsson elaborates:
Most importantly, when individuals, based on their extensive experience and reputation, are nominated by their peers as experts, their actual performance is occasionally found to be unexceptional. For example, highly experienced computer programmers’ performance on programming tasks is not always superior to that of computer science students … and physics professors from UC Berkeley were not always consistently superior to students on introductory physics problems … More generally, level of training and experience frequently has only a weak link to objective measures of performance. [7]
Research shows that once professionals reach phase three, the remainder of their professional career requires only stable performance, which can be achieved with minimal effort. For this reason, beyond the first two years in a new profession, experience correlates weakly with performance.
So how can you build a skill-set faster than your peers and continue to surpass them over time? Deliberate practice is the answer. The Handbook offers definitions of deliberate practice, but I like the way Joshua Foer explains it best in Moonwalking with Einstein:
“What separates experts from the rest of us is that they tend to engage in a very directed, highly focused routine, which Ericsson has labeled ‘deliberate practice.’ … They develop strategies for consciously keeping out of the autonomous stage while they practice by doing three things: focusing on their technique, staying goal-oriented, and getting constant and immediate feedback on their performance. In other words, they force themselves to stay in the ‘cognitive phase.’”
“Amateur musicians, for example, are more likely to spend their practice time playing music, whereas pros are more likely to work through tedious exercises or focus on specific, difficult parts of pieces. The best ice skaters spend more of their practice time trying jumps that they land less often, while lesser skaters work more on jumps they’ve already mastered. Deliberate practice, by its nature, must be hard.” [8]
Sidenote: Moonwalking with Einstein is the most entertaining introduction to this topic. If you need a book to get you started, I recommend it.
If you want to truly excel at something, how you spend your time preparing is what makes the difference. The amount of time is of little consequence if the training does not require concentration and effort. In fact, Ericsson writes that the reason most frequently provided by expert performers for the amount of effective training achieved per day is “an inability to sustain the level of concentration that is necessary.” [9] For this reason most expert performers never train for more than 4 to 5 hours per day.
What does this mean for ASimpleModel? Watching the videos and reading the notes are not sufficient to learn this skill set, and certainly not sufficient to gain proficiency. You must engage in deliberate learning exercises. Build the models yourself. Initially you may want to follow along with the video instruction, but once that becomes easy, look for new examples that will test your limits. ASM provides multiple exercises, but I would also encourage taking what you learn and applying it to new context. Download annual filings and build your own models. The process will expose areas of weakness, and highlight which topics you need to revisit.
TESTING & FEEDBACK:
Effective learning requires awareness of what you do not know, and feedback on your performance. The simplest explanation I found came from the book Superforecasting. The author asks you to imagine the process of a basketball player attempting to improve his free throw by practicing with the lights off. To the player, a perfect shot will sound the same as a shot below the rim that brushes the net. Without feedback, in this case provided by simply turning the lights on, he will never improve.
As a more elaborate example, consider the following:
“…it’s been found that in a few fields of medicine, doctor’s skills don’t improve the longer they’ve been practicing. The diagnostic accuracy of professional mammographers, for example, doesn’t get more accurate over the years. That’s because mammographers usually only find out if they missed a tumor months or years later, if at all, at which point they’ve forgotten the details of the case and can no longer learn from their successes and mistakes.”
“One field of medicine in which this is definitively not the case is surgery. Unlike mammographers, surgeons get better with time. What makes surgeons different from mammographers, according to Ericsson, is that the outcome of most surgeries is usually immediately apparent – the patient either gets better or doesn’t – which means that surgeons are constantly receiving feedback on their performance.” [10]
If all you do to prepare is watch instructional video and read notes you will have no means of measuring your performance. On ASimpleModel you can generate feedback two ways:
- Work through the exercises and build the models.
- Work through the quizzes and evaluate your results.
The former has been discussed in this post already, but I will provide one more example to demonstrate how deliberate practice and generating feedback can overlap. The best indicator of a chess player’s skill set, according to Ericsson, is not the amount of tournament play, but the amount of time spent studying published games between the best chess masters in the world.
“They typically analyze the games by playing through the games, one move at a time, to determine if their selected move matches to the corresponding move originally selected by the masters. If the master’s move in the studied chess game differed from their own selection, this would imply that their planning and evaluation must have overlooked some aspect of the position.” [11]
Ericsson continues by pointing out that this process permits the chess student to evaluate precisely where he / she went wrong.
With respect to the second option listed above, the objective of the quizzing platform is to change the way users think about testing. This is not a one-time exercise used to evaluate proficiency, but a process that permits users to identify areas of weakness. As an ASM subscriber you control who can see your scores, and whether or not you want to share them at all. You are also permitted as many attempts as you like, and in fact multiple attempts are encouraged. Per the authors of Make it Stick:
“But if we stop thinking of testing as a dipstick to measure learning – if we think of it as a practical retrieval of learning from memory rather than ‘testing,’ we open ourselves to another possibility: the use of testing as a tool for learning.” [12]
That is the objective at ASM. This platform should be used to create greater self-awareness and focus learning efforts on areas of weakness.
“The act of retrieving learning from memory has two profound benefits. One, it tells you what you know and don’t know, and therefore where to focus further study to improve the areas where you’re weak. Two, recalling what you have learned causes your brain to reconsolidate the memory, which strengthens its connections to what you already know and makes it easier for you to recall in the future. In effect, retrieval – testing – interrupts forgetting. Consider an eighth grade science class. For the class in question, at a middle school in Columbia, Illinois, researchers arranged for part of the material covered during the course to be the subject of low-stakes quizzing (with feedback) at three points in the semester. Another part of the material was never quizzed but was studied three times in review. In a test a month later, which material was better recalled? The students averaged A- on the material that was quizzed and C+ on the material that was not quizzed but reviewed.” [13]
The authors of Make it Stick detail the first-hand research described above, and simultaneously offer several examples of prior studies:
“A second landmark study, published in 1939, tested over three thousand sixth graders across Iowa. The kids studied six-hundred-word articles and then took tests at various times before a final test two months later. The experiment showed a couple of interesting results: the longer the first test was delayed, the greater the forgetting, and second, once a student had taken a test, the forgetting nearly stopped, and the student’s score on subsequent tests dropped very little.” [14]
“In 1978, researchers found that massed studying (cramming) leads to higher scores on an immediate test but results in faster forgetting compared to practicing retrieval. In a second test two days after an initial test, the crammers had forgotten 50 percent of what they had been able to recall on the initial test, while those who had spent the same period practicing retrieval instead of studying had forgotten only 13 percent of the information recalled initially.” [15]
“In 2010 the New York Times reported on a scientific study that showed that students who read a passage of text and then took a test asking them to recall what the read retained an astonishing 50 percent more of the information a week later than students who had not been tested.” [16]
CONCLUDING REMARKS:
The theme I continue to encounter, whether expressed as deliberate practice, testing or feedback, is that effective learning requires effort and doesn’t always feel natural. Students prefer rote drills and rereading because it’s easier, but unfortunately it is not nearly as effective. From what I have read, and while it is not explicitly introduced as a “learning mindset,” my favorite description of what learning should feel like comes from Daniel Kahneman’s book Thinking Fast and Slow.
Kahneman does not speak directly to the learning process in the sequence that follows, he is instead focused on the differences between System 1, which operates automatically and with little effort, and System 2. Kahneman introduces System 2 by asking you to look at a multiplication problem (17 x 24), and describing what happens when you attempt to solve it:
You first retrieved from memory the cognitive program for multiplication that you learned in school, then you implemented it. Carrying out the computation was a strain. You felt the burden of holding much material in memory, as you needed to keep track of where you were and of where you were going, while holding on to the intermediate result. The process was mental work: deliberate, effortful, and orderly – a prototype of slow thinking. The computation was not only an event in your mind; your body was also involved. Your muscles tensed up, your blood pressure rose, and your heart rate increased. Someone looking closely at your eyes while you tackled this problem would have seen your pupils dilate. Your pupils contracted back to normal size as soon as you ended your work – when you found the answer (which is 408, by the way) or when you gave up. [17]
Building your own financial models, at least initially, should feel like the description above. And while it doesn’t feel as good as System 1, just remind yourself that it is the most effective means of learning the skill set faster. Concentration and difficulty are essential to learning.
I think the most powerful message I can leave you with, is that regardless of your approach to learning, your biggest advantage is that it is rare for people to make the commitment. As the research suggests, most people learn enough to get by, and then abandon the learning process. If you can commit to continued learning, you have an advantage over most. And if you ever think to yourself, “some people are just naturals, why should I bother?” Know that it’s not true:
“Even the well-known fact that more ‘talented’ children improve faster in the beginning of their music development appears to be in large part due to the fact that they spend more time in deliberate practice each week.” [18]
“Until most individuals recognize that sustained training and effort is a prerequisite for reaching expert levels of performance, they will continue to misattribute lesser achievement to the lack of natural fits, and will thus fail to reach their own potential.” [19]
Now, if the skill is basketball, then yes, there are advantages you cannot work around. Seven feet tall is seven feet tall no matter how much deliberate practice is involved. Fortunately, height won’t help you build models. As Ericsson writes in the introduction, “Not even IQ could distinguish the best among chess players or the most successful and creative among artists and scientists.” [20] The playing field is even. Get started.
Footnotes:
[1] Roediger III, Henry L. “How Tests Make Us Smarter.” New York Times 18 July 2014. Print
[2] Ibid.
[3] Ibid.
[4] Joshua Foer, Moonwalking with Einstein (New York: Penguin Books, 2012), p. 208
[5] Ibid., p. 45
[6] Peter C. Brown, Henry L. Roediger III, Mark A. McDaniel, Make it Stick: The Science of Successful Learning (Cambridge: The Belknap Press of Harvard University Press, 2014), p.
[7] Ericsson, K.A. The Cambridge Handbook of Expertise and Expert Performance. (Cambridge, England: Cambridge University Press, 2006), p. 686
[8] Joshua Foer, Moonwalking with Einstein (New York: Penguin Books, 2012), p. 171
[9] Ericsson, K.A. The Cambridge Handbook of Expertise and Expert Performance. (Cambridge, England: Cambridge University Press, 2006), p. 699
[10] Joshua Foer, Moonwalking with Einstein (New York: Penguin Books, 2012), p. 173
[11] Ericsson, K.A. The Cambridge Handbook of Expertise and Expert Performance. (Cambridge, England: Cambridge University Press, 2006), p. 697
[12] Peter C. Brown, Henry L. Roediger III, Mark A. McDaniel, Make it Stick: The Science of Successful Learning (Cambridge: The Belknap Press of Harvard University Press, 2014), p. 19
[13] Ibid., p. 20
[14] Ibid., p.31
[15] Ibid., p.31
[16] Ibid., p. 29
[17] Khaneman, Daniel Thinking Fast and Slow. (New York : Farrar, Straus and Giroux, 2011), p.
[18] Ericsson, K.A. The Cambridge Handbook of Expertise and Expert Performance. (Cambridge, England: Cambridge University Press, 2006), p. 692
[19] Ibid., p. 699
[20] Ibid., p. 10