Super Forecasters: Super Excuses. On The Art Of Being Less Wrong.
When it came to the Brexit vote and the election of Trump, the so-called Superforecasters, were, almost to a man, found to be guilty
of selectively sampling information in order to confirm their own pre-exisiting hypothesis. They had collectively fallen in to the trap of believing that they
somehow had access to information that was not available to anyone else. And, despite the stories that they told themselves, their students and their
Twitter followers about their professed ability to debias and to embrace Bayes optimality - they actually provided the greatest known example of confirmation bias and motivated reasoning in the Post Great Recession Era. Perhaps the likes of Gadsby and Tapping will, in the light of these findings, see fit to extend their research on Bayes
Optimality to financial and betting markets.
When reviewing why the so-called Superforecasters produced such poor predictions when it came to the 2016 U.S. Presidential election, one would do well to recall that Tetlock had previously found that people who spend their time and earn their living, studying a particular topic, produce poorer predictions than dart throwing monkeys.
In Expert Political Opinion, Tetlock reported that the average expert's predictions were no better than a random guess; moreover, he also found that there is something about being a high ranking expert that interferes with ones ability to forecast. Human affairs, it is noted, are mostly random and intractable, because as people we are prone to unpredictable behaviour: Tetlock's Superforecasters got Brexit wrong and Tetlock's Superforecasters got the 2016 U.S. Presidential Election wrong. They were not so super after all.
Should any of this come as a surprise - of course not. It has long been known that better educated, analytically minded people, more often than not display a strong propensity towards behaviors such as ignoring contradictory evidence, compartmentalizing knowledge, and generally deluding themselves that they are the sharpest nail in the box.
Following on from the outcome of the 2016 U.S. election Tetlock took to Twitter to say; we will be tallying the conventional-wisdom casualties of tonight for many years. No doubt, in his next book Tetlock will be telling us why it is that superforecasters get things wrong, but only less wrong, and how what differentiates them from the rest of us is their ability to take it on the chin; go back to the drawing board; recalibrate; put into practice the Bayesian brain etc.
The so-called super forecasters and supporters of the notion of the conventional wisdom, have one thing in common - when things go badly wrong, they are quick to come up with the excuses; timing, the occurence of an unforseen event, or, perhaps, they were simply wrong but for the right reasons, or, indeed, as was the case with Nate Silver following the 2016 U.S. Preisdential Election, perhaps they were simply less wrong. Hindsight bias is commonplace; I knew it all along or perhaps, I was simply too clever by half for my own good. What is known in sports betting circles as after-timing. Self-promoting begets self-promoting; if you screw up with the initial prediction, there is always a second bite at the cherry - the explanation/excuse as to why you screwed up the prediction in the first place. And should you be lucky enough to have the label superforecaster attached to your person, you can always rest happy with the notion that you were simply less wrong than all of the other so called superforecasters.
Self-professed Bremain supporter and media puppet Leighton Vaughan Williams, was quick to dismiss the collective intelligence of 17.4m Brexit voters, and to ignore the complex interrelationship of factors that had contributed to the vote, when in a newspaper article in the wake of the result he declared that it was the Sun and Mail wot won it. Many asked, quite correctly, why it was that he had not shared this pearl of wisdom with his devoted Twitter followers on the day before the vote had taken place.
In the wake of the 2016 U.S. Presidential Election it was claimed, by many members of the commentariat that Facebook had swayed the outcome of the election through, according to the Guardian allowing misinformation to spread unfettered on its network, skewing people’s perceptions. The common theme - the people were duped - but the experts failed to see it coming.
Poll supremo Frank Luntz, who famously took to Twitter to declare that Hilary Clinton will be the next President of the United States later went on to declare that the polls had been incorrect because Trump voters refused to participate, because they didn't want to help the media/establishment.
The statistician and survey scientist Drew Linzer, another who got it badly wrong, took to Twitter to say (one hopes with a sense of humility and indeed humour) There's no sugarcoating that my forecast of a Trump loss with 215 EVs is much lower than the 306 he will likely end up with. Save that one for the grandchildren Drew.
For Sam Wang (whose organisation ascribed a totally ridiculous 99% probabilty to a Hillary Clinton victory!) and Nate Silver, who got it wrong but less wrong, it was the late-deciding voters that delivered victory to Trump.
In the run up to the election, Mona Chalabi, one of the few to correctly call the election for Trump, highlighted the fact that there existed amongst the U.S. electorate, a class of voter, irrespective of gender or race, typically, but not exclusively amongst the poorest in society, that distrusted professionals, and that would, accordingly, not be willing to reveal their voting intention to pollsters.
Chalabi was of course right, but she lacked the self-promotion skills of the Nate Silver machine, and she was also a woman, operating in a world of mainly male pollsters. Writing in the Guardian, Chalabi later broke ranks, and spilt the beans on superforecaster Nate Silver and his organisation, for whom she had once worked, levelling charges of subjective bias and arrogance at them.
Writing in the FT Philip Delves Broughton, spoke of the psychology of a particular class of Trump voter, who, he said, were reluctant to share their voting preferences with a stranger, and were, more often than not, more forthcoming when sharing their intentions with a recorded service, than with a live voice on the end of the phone.
The real smoking gun, in this regard, can be found in an article written on Nate Silver's FiveThirtyEight website entitled Live Polls And Online Polls Tell Different Stories About The Election by Harry Enten (posted on the 31 August 2016.)
In concluding his piece, which showed a very real disparity between live and non-live polls, Enten wrote; I’d put more faith in the live-interview polls than in other types of surveys, all else being equal. Indeed, our forecast models do just that.
The article clearly demonstrated that some voters were affected when it came to engaging with a live voice on the end of the phone, but as was the case with many analysts vis a vis the Brexit vote, it reached the wrong conclusions. Enten had put his faith in the live-interview polls, when in reality it was the online polls that were revealing the voters' real intentions. The notion of a Brexit and a Trump presidency were highly contentious, devisive issues, and people simply did not want to be associated with the negative connotations that formed the centrepiece of the liberal media narrative (if you voted Trump/Brexit you were racist etc...).
Trump was portrayed as being a racist, misogynist, bully, whilst in the UK the death of Jo Cox had cast a long shadow.
Clinton herself had said that Trump’s supporters were irredeemable, deplorables characterized by implicit racism and during the
second presidential debate she openly accused him of being Putin's stooge.
Both in relation to Brexit and the 2016 U.S. Election acrimony was the order of the day.
Clinton as seen performing better in live interview polls
The following table shows that five of the eleven polls released in the immediate run up to the Brexit vote had the Leave side winning. Four of them were online polls. A significant aspect of the narrative fallacy spun by those that opposed Brexit, was, that given that a majority of the polls that came out in favour of Brexit were online polls, they had to be wrong - people, it was held, were more likely to reveal their true voting intention when contacted by telephone (paradoxical logic par excellence and the very mistake that Harry Enten, and many others were to make as regards the outcome of the 2016 U.S. election.).
Breixt had provided a clear clue as to what happens to polls when public opinion is highly polarised and there is a stigma surrounding ones voting preference: but the U.S. pollsters and analysts, almost to a man, chose to ignore the Brexit lesson (their election was different they said, it had a history).
The prevailing narrative following the Brexit vote was of course that the disenfranchised white working class had revolted against the metropolitan elites. In fact, most leave voters were in the south: the south-east and the south-west; some of the most affluent areas in the country. In the US post-election reporting typically covered the role of white working-class or college-educated voters in the election, but research from Pew very clearly demonstrated that the middle class had shifted allegiance over the course of Obama’s two terms as president; The Republican Party made deep inroads into America’s middle-class communities in 2016. Although many middle-class areas voted for Barack Obama in 2008, they overwhelmingly favored Donald Trump in 2016, a shift that was a key to his victory.
Filipe Campante and David Yanagizawa-Drott In short, places with higher social mobility have tended to vote more Republican than places with lower social mobility. They also voted more heavily for Trump than for Romney four years ago.
A significant consequence of the 2016 U.S.Presiential Election was that the term prediction markets as had been attached to conventional betting markets/betting exchanges, by the likes of Chris.F.Masse, was finally laid to rest. The prediction market concept had embraced the rather naive notion that markets are omnipotent and capable of pricing in all of the available information (their probabilities are deterministic). They had come into the U.S. election badly bruised, following their total failure to predict the outcome in the 2015 UK General Election, the Greek Referendum and the Brexit vote (or perhaps they were just less wrong!). The following table shows the Betfair betting exchange betting market on the night of the 2016 U.S. election. Hillary Clinton is seen trading at 1.14, which tanslates into an implied probability of 86%.
In betting on individual States, the traders on the Betfair betting exchange, taking their steer from the polls, had Clinton nailed on in Florida, Michigan, Pennsylvania and Wisconsin. She lost them all to Trump, and it was these states that swung the election for Trump. State polls had systematically underestimated Trump's standing in these states, and this oversight then fed into the super bullish implied probabilities that were attached to Clinton chances vis a vis winning the Presidency: 85% at the New York Times, 71% at FiveThirtyEight and 99% at the Princeton Election Consortium etc...
One of the chief advocates of the term prediction markets , Justin Wolfers, was left with a significant amount of egg on his face in the wake of the U.S. election result.
In the run up to the election, Wolfers had taken to Twitter to take a swipe at Nate Silver, both for underestimating Hillary Clinton's chances and for running more than two models on the event, suggesting that this could give rise to the temptation to point to one of them ex-post.
Wolfers also suggested, that in the light of the high probabilities ascribed to Clinton's chances in other polls, that Silver's model must be wrong.
As the table below reveals, at the time of Wolfer's attack, Nate Silver's organisation had Clinton with an implied probability of 86%!
Wolfers also wrote an academic paper with Eric Zitzewitz, in which the pair claimed that a Trump win in the election would tank the financial markets: Market participants quoted in the financial press suggest an apparent consensus that a Trump victory would lower equity prices, weaken the economy, and increase risk. Our calibration of these movements using prediction market movements during the debate suggest that the magnitudes of these effects would be much larger than in past Presidential elections. The estimated magnitudes of the Trump discount are more comparable to those that accompanied the Brexit vote or 2003 Iraq War. As per, Howard Marks of Oaktree Capital; ... the U.S. stock market had its best week since 2014! The Dow Jones Industrials rose almost 5% for the week, taking them to a new all-time high. The Dow was up every day last week. It rose on Monday and Tuesday, when Clinton was expected to win. And then it rose Wednesday, Thursday and Friday, after she had lost.
The New York Times, for which Wolfers occasionally writes, and which has been rightly lambasted for its extermely jaundiced coverage of the election, embraced the prediction business itself during the election, happily advertising a model that ascribed an 86% probability to a Clinton presidency.
During the primary elections, Bernie Sanders labelled Clinton "corrupt" and assailed her for taking large speaking fees from big business and the financial industry. It is highly likely that Sanders wounded Clinton, and that some of his more staunch supporters either chose to stay away from the polling booths, or instead, turned towards Trump, who they saw not as a traditional Republican, but as an alternative anti-establishment candidate who was prepared to take on corporate America. This would account for some of the five million drop in numbers between those that had voted for Obama in 2012 and those that voted for Clinton in 2016.
Writing in the FT Philip Delves Broughton wrote that Mrs Clinton’s team so strongly believed their own internal models, that told them that Wisconsin was a shoo-in, that Clinton did not visit that state between securing the nomination and election day (even though the state had a Republican governor, Legislature, and Attorney General.) Sanders had won the Wisconsin primary convincingly and it is interesting to note that turnout was down 6.6 per cent to a 20-year low for the 2016 presidential election. Many of Sanders' supporters had clearly stayed at home.
Trump bettered the Real Clear Politics average by 7.5 points in Wisconsin and 4 points in Michigan.
In the London Review of Books, R.W. Johnson (Trump: Some Numbers) noted that during the campaign Debbie Dingell, the Democratic Congresswoman for Michigan’s 12th district;
repeatedly warned Clinton (whom she supported) that Michigan was not safe and that Trump could win. People thought she was nuts: The auto workers went heavily for Sanders, who won the primary. From that moment on Dingell feared that they – and Michigan – would go for Trump, and they did.
The Clinton campaign strategy towards Pennsylvania, which centred around having the likes of Katy Perry, Springsteen and Bon Jovi
play concerts in Philadelphia, was never going to win over the more hardline amongst Bernie Saunders supporters. In Philadelphia there were almost 100,000 fewer voters compared with 2012. The exit polls for Pennsylvania showed Clinton significantly underperforming Obama among voters under the age of 30 (a key Sanders demographic).
It was claimed in the media that around 10 percent of Sanders' supporters would reject Clinton, vote for a third-party candidate or,indeed cast a vote for Donald Trump. This number was more than likely conservative, but such a stance, if carried through, may well have accounted for Clinton's defeat in Florida.
It is highly likely, that many of the lower income voters that voted for Sanders in the primaries, and for Obama in 2012, did not vote for Clinton in 2016. It is almost certain that a significant number of Sanders' supporters under the age of 30 also did not vote for Clinton. And, many may instead have choosen to vote for third-party candidates. The reopening of the Hillary Clinton Email Investigation by FBI Director James Comey, perhaps crystalised their opposition to Hillary Clinton and gave them a further reason to stay away from the polls. In Michigan and Wisconsin Clinton received around 300,000 fewer votes than Obama.
The total number of votes cast for Sanders in the Florida, Michigan, Pennsylvania and Wisconsin primaries had been 2,469,855.
During the 2016 Presidential Election, only 216,510 votes had separated Donald Trump and Hillary Clinton across these four states.
To become president, Clinton did not even have to win Florida; if 51,799 voters (that is 8% of those that had voted for third-party candidates) in Michigan, Pennsylvania and Wisconsin had changed their minds and handed those states to her, then there would be no President Trump.
What it Would Have Taken to Flip Four Battleground States
Trump's vote lead
Votes for third-party candidates
% of third-party vote Clinton needed to win
S: Election results from the Atlas of U.S. Presidential Elections
Wisconsin Democratic Primary
Wisconsin Presidential Election
Michigan Democratic Primary
Michigan Presidential Election
Florida Democratic Primary
Florida Presidential Election
Pennsylvania Democratic Primary
Pennsylvania Presidential Election
S: Election results from the Atlas of U.S. Presidential Elections
The truth is, of course, that there is no simple binary explanation as to why Hillary Clinton lost the election - if it were that easy, the polls may, indeed, have even got it right. Voting patterns always shift, and it is next to impossible to extract one single pattern and say that that was the one that caused a defeat. In a nutshell, what we can say with some degree of certainty is that fewer African Americans and Latino women went for Clinton in 2016 than had voted for Obama in 2012, and,moreover, there is evidence, from the likes of Edison Research that somewhat contrary to the popular narrative, Trump equalled or bettered Romney's performance among Latinos generally. Between 2004 and 2012, Obama had made big gains in relatively Hispanic areas; in 2016, Clinton did not. White men with only high school education did support Trump and did make a significant contribution towards swinging Michigan, Pennsylvania and Wisconsin to him, but it is also the case that Trump performed better in the suburbs in these states than Romney had. Joel Kotkin notes that; Trump won suburbia by a significant five percentage point margin nationally, improving on Romney’s two-point edge, and by more outside the coastal regions.....Donald Trump won 50 midwestern electoral votes that went to Barack Obama in 2012 — Iowa, Wisconsin, Michigan, and Ohio — plus 20 more in Pennsylvania. Trump won more voters than Romney in MI, PA, FL, NC, WI, IA, OH, NH. Clinton's voters stayed at home; take your pick, create your own narrative (as per below).
What we can say with certainty is that the polls and the analysts were guilty of what the late Paul Watzlawick called insufficient reality adaptation. They had, along with the Clinton camp, significantly underestimated the degree and extent to which Bernie Sanders had wounded Clinton, and the degree and extent to which his suporters would either refuse to vote for her or would switch to Trump, of the other third-party candidates. They had also significantly underestimated the strength and importance of the white vote. Most Americans are white, most are Christian, most don't have college degrees, and it had been clearly demonstrated by The Upshot that millions more white, older working-class voters went to the polls in 2012 than was found by exit polls on Election Day. Moreover, the white vote had been moving away from the Democrats in the Rust Belt for the past twelve years, and this is something that should have been factored in to the models. Autor, Dorn, Hanson and Majlesi (2016 et al) had written extensively on the fact that trade shocks had political implications, and they clearly demonstrated that the consequence of an increase in import competition from China, was a substantial shift to the right in those states that were most affected. The pollsters and analysts had also, as was the case with the Brexit vote, shown themselves to be simply incapable of factoring into their models the impact and outcome of what was a highly contentious, divisive and acrimonious campaign. This was not about about the age old conventional reasons of misunderestimating the size of the Republican base, and the way in which it coalesced at the end of the campaign. This was about misunderstanding human psychology; of not getting the enthusiasm and eclectic nature of Trump's supporters, and the impact that they would have upon the outcome of the election, in the face of a weak democratic candidate, who was already wounded by her own side, and who displayed a degree of myopic arrogance vis a vis the states that actually mattered when it came to winning the election. The nickname Crooked Hillary stuck - it, Comey's late intervention, and Bernie Sanders, combined, gave people a reason to stay at home, which they duly did in their millions. It can also be argued that the overly optimistic polls and prediction markets also contributed to this phenomena, giving, as they did, the false impression that the race was a shoe-in for Hillary Clinton. If they had said that Trump was a 99% chance then he too may have lost the election.
The polls and models failed for a very simple reason; because they did not take non-stationarity into account. They had ignored the important lessons that had been thrown up by the Brexit vote, vis a vis voter psychology, and they had failed to understand and come to terms with the underlying social and cultural shifts that allowed both Brexit and Trumpism to win the day. They had treated politics like it was a physical science, ignoring Vilfredo Pareto’s assertion that the
foundation of political economy and, in general, of every social science, is evidently psychology.
In assigning near Black Swan status to a Trump victory, followers of the so-called conventional wisdom were exposed for being nothing more than a bunch of chancers, who had happened to get lucky on occasion in the past. The large-scale public polling failure served to expose the polling company model makers as a group of people, who, blinded by their own mathematical models, were incapable of believing anything other than that the future would be like the recent past (survivorship bias),and, revealed them as being people happy to embrace the limelight when they are right, and quick to take to the shadows when their models fail to incorporate unquantifiable surprises (or, as in the case of Nate Silver, to turn things around by basking in the glory of being less wrong).
To cite this article: Niall O'Connor "Super Forecasters: Super Excuses. On The Art Of Being Less Wrong." (Published on Bettingmarket.com 2017. All Rights Reserved.)