Philip Tetlock spent 30 years exploring what makes someone a superforecaster (not the ones you see on TV). In his book Superforecasting, he distills his work into several commandments on how we can improve our judgment without any complex algorithm.
1. Triage: Focus on questions where your hard work is likely to pay off
Tetlock
wrote “Don’t waste time either on easy questions (where simple rules of
thumb can get you close to the right answer) or on impenetrable
questions (where even fancy statistical models can’t beat the
dart-throwing chimp). Concentrate on questions in the Goldilocks zone of
difficulty, where effort pays off the most.”
In
investing, you want to spend a majority of your time on the 2 to 3 most
important variables that determine 80% of the outcome. Why only 2 to 3
variables? Because success rate falls exponentially when an investment
has many moving parts. If an investment idea requires 6 variables to
work out to be profitable, the success rate quickly plummets to 53%
(0.9^6) even when you’re confident that each variable carries a 90%
likelihood of happening.
You
also want to avoid questions that are too hard to answer. Let’s say you
are researching an O&G company. You figure that the oil price is an
important variable but what’s your probability of predicting it
correctly 3 to 5 years out? That’s an impenetrable question where you
won’t do much better than flipping a coin. Instead, another important
yet solvable variable is to find out the cost structure of the company
and its sustainability. Given that most O&G players are a price
taker, the most efficient company sitting at the bottom of the
variable-cost curve will reap the most benefits regardless of the oil
price level.
2. Breaking seemingly intractable problems into tractable sub-problems
George
Polya, a Hungarian mathematician, once advice “If you can’t solve a
problem, then there is an easier problem you can solve: find it.”
Is
this stock a buy? That is a big hairy problem. One way to solve it is
to break it into sub-problems. Every time you break down a big problem,
you’re asking “What needs to happen for this to be true?”. In this case,
what needs to happen to qualify this stock as a buy? Let’s say you
break it into 3 sub-problems:
Market valuation
Business characteristics
Portfolio hurdle rate
You
can further break each of this sub-problem into more sub-problems as
necessary until you can solve it. If you’re looking at the business
characteristics, you can break that further into competitive advantage,
financial strength, industry dynamic and so on. Once you’re able to
solve those sub-problems, you can begin to move your way up by solving
the bigger problem right above and that will eventually lead you back to
answer “Is this stock a buy?”
3. Strike the right balance between inside and outside views.
The
inside view looks at specific circumstances surrounding a situation,
while the outside view tied circumstances to an appropriate reference
class by asking what happens when others encounter something similar?
It
is easy to engross in the specifics surrounding a company, such as the
growth story and extrapolate based on what we see. But a good forecast
also requires the outside view or base prediction. If a company is
forecasted to grow 15% annually over the next 5 years (inside view),
while its competitors have a growth rate closer to 9% (outside view),
then you need a good reason to explain the difference. More often than
not, your revised growth rate will likely fall somewhere in between
those numbers. We shouldn’t dismiss the inside view entirely, of course.
What’s unique to a company i.e culture, could sometimes turn out to be a
good predictor of success. But at the same time, the outside view tames
overconfidence and avoid base rate neglect so we don’t miss the forest
for the trees. As a rule of thumb, start from the outside view before
adjust towards the inside view to avoid anchoring bias and
overestimation.
4. Strike the right balance between under- and overreacting to evidence.
If
a company reported two consecutive lower quarterly earnings due to a
challenging environment, should you take that as a canary in the coal
mine for more bad news or just a temporary setback that is unlikely to
affect the company’s future? There’s no easy answer because there is a
lot of unknown. But one way to weigh evidence appropriately is to use
Bayes rules to update belief. Tetlock wrote, “The best forecasters tend
to be incremental belief updaters, often moving from probabilities of,
say, 0.4 to 0.35 or from 0.6 to 0.65, yet superforecasters also know how
to jump, to move their probability estimates fast in response to
diagnostic signals.”
To apply Bayes rules to this scenario, you need 3 things:
Prior
probability. This is the initial estimate on what probability would you
assign to the business’s earning power being impaired before the
company reported 2 consecutive lower earnings. Let’s say given the
company’s strong track record, you estimate there is a 15% probability.
Put it another way, you believe there is an 85% probability the business
is going to do well in the long-term. (Hence you bought the shares).
Next,
we need the probability that the hypothesis being true—2 consecutive
lower quarterly earnings is indeed a sign that the earning power is
impaired. Let’s say based on your understanding of the company’s
competitive position in the market, you think it is unlikely. But at the
same time, the fact that the company has never reported any lower
earning prior to this event is a cause of concern. You place the
probability for the hypothesis to be true at 50%.
Last,
the probability that the hypothesis is false. If the earning power
remains intact, what could explain those lower earnings? It could be a
shift in product mix to lower margin items, lower volume sold due to
supply constraints and so on. Let’s say you put this at 28% after
considering all alternative outcomes.
Now,
we can establish a posterior probability—how likely the earning power
is impaired given there are two consecutive lower earnings? We can see
the probability has increased from the initial 15% to 24% after two
consecutive lower earnings. So next time when you receive another piece
of new evidence, 24% will become the prior probability (base rate) to
update your belief.
5. Look for the clashing causal forces at work in each problem
For
every good argument, there is always a counterargument. Your role as an
investor is to synthesize all the positive and negative evidence and
make a decision. If a company won a multi-million dollar contract which
improves the future prospect of the business, also think about the other
side of the argument on what can be negative: prolonged delay in
project completion, cost overrun, contract termination, unattractive
return etc. If you believe investing using margin is risky, think about
circumstances when it can be useful. If you’re a true believer in owning
only quality companies in a concentrated portfolio, consider
circumstances when this strategy becomes unfavorable, and vice versa if
you prefer owning cheap mediocre stocks. This can be applied to any
general argument from operating leverage, company debts, profit margin
to situation specific to an investment.
The lesson here is you have to have the ability to hold contradictory information in
your head to improve forecasting skills. All great investors are great
synthesizer—the ability to blend both sides of things into a single
worldview and make decisions accordingly.
6. Strive to distinguish as many degrees of doubt as the problem permits but no more
If you say “I’m fairly confident that this company can increase its profit over the next 3 years.”, what does fairly mean?
If the company downgraded their earnings 6 months later, would you
still remember how you feel about the investment 6 months prior? And do
you reduce your fairly to likely, possibly or something else? Using
words is not informative, not to mention you probably have no idea what
that word means as time goes by. Even if you wrote it down.
A
better way is to translate them into numeric probabilities. If you
think the probability of an increase in profit over the next 3 years is
75%, that would be the prior probability. And when you receive the news
of a profit downgrade, you can use Bayes rule to revise and update your
belief. Thinking in probability also allow you to think in expected value and alternative outcomes, which are the first step to think clearly and improve accuracy.
7. Strike the right balance between under- and overconfidence, between prudence and decisiveness
Good
forecasting requires a good calibration between confidence and
accuracy. If confidence gets ahead of accuracy, you risk going into an
investment that falls outside your circle of competence, where you think
you know when in fact you don’t know; If accuracy outruns confidence,
you risk missing out on great opportunities by pondering too long. A
good calibration means know when to sit down and do nothing as well as
knowing when to swing for the fences. A good grasp on the edge of your
circle of competence is critical to improving calibration, and an easy
way to do that is to read a lot.
8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases
Learn
from your mistakes is a great way to improve forecasting. Mistakes are
not necessarily ones you’ve made a loss; you can make a big gain through
pure luck. So it is critical to look at all of your investments and
focus on the thought process rather than the outcome. This underlines
the importance of having a decision journal so you can refer back to
your decision-making process. It also prevents hindsight bias.
9. Master the error-balancing bicycle
The
best way to become a better forecaster is to practice it. But first,
you need to create a feedback loop so you can learn and improve as you
go along. A good feedback loop should have these criteria:
Time-specific. If you make a forecast without a timeframe, your forecast is untestable. Set a date to score your forecast.
Quantifiable.
Translate words into numbers whenever possible. “83% probability”
instead of “very likely”; “22% ROC” instead of “high ROC”. Always ask if
your forecast will leave any room for interpretation or arguments. As a
general rule, it is more useful to forecast business performance than
the share price.
Measurable.
You can’t improve what you don’t measure. Brier score is a good tool to
measure the accuracy of your forecast. The lower the score, the more
accurate you are. Superforecaster tends to have a Brier score of around
0-0.25.
