Risk intelligence in Baalbeck

Last week I went to see the Roman ruins in Baalbeck, Lebanon. I had wanted to visit Baalbeck ever since I arrived in Beirut two months ago, but had hesitated about going because I’d heard that the security situation there was dicey. Last year, seven Estonians were kidnapped in while cycling near Baalbeck, in the Bekaa Valley. Last month, the US Embassy in Lebanon sent an emergency message warning Westerners not to travel to the Baalbeck area because of clashes between the Lebanese authorities and local criminal groups. The latest clash erupted only last week, when three Lebanese Army soldiers were wounded during a shootout early Tuesday in the town of Baalbeck itself.Dylan in Baalbeck

Browsing online, however, I came across various traveller’s bulletin boards that told a slightly different story. One tourist noted, on 14 March, that they were with some friends in the Baalbeck ruins early the previous Sunday morning, when suddenly they heard lots of automatic arms fire in nearby, and also “heard what sounded like something whizzing across the sky above us, which was followed by a few very large explosions.” However, it seemed to start up and die off quickly and all at once so they assumed it was a military drill of some sort. The travellers didn’t seem that scared.

It’s important to take reporting bias into account when reading newspapers and searching online for security information. Journalists don’t file reports saying “nothing happening round here,” and tourists are far more likely to post something online when they hear gunshots than when they don’t. So it’s hardly surprising that a Google search for “Baalbeck warnings dangers” will turn up a bunch of scary stories.

One way of getting a better idea of the security situation somewhere new is simply to ask people who’ve been there recently. This way, you avoid giving too much weight to the self-selecting sample of those people who go to the trouble of writing about stuff. So last week I asked everyone I knew in Beirut if they had been to Baalbeck recently, and whether they thought it was dangerous. The overwhelming consensus was that it was fine.

Weighing up all the evidence I had gained from newspapers, online reports, and gossip, and taking into account the various biases that might distort these sources, I tentatively concluded that my chances of being shot or kidnapped in Baalbeck were minimal – low enough to constitute an acceptable risk. So my sister and I headed out there in one of the rickety minibuses that are the cheapest way of getting around in Lebanon, and for a few hours the following morning, we had the ruins to ourselves.

Risk intelligence and creativity

Over the past few days a number of fashion designers have started following me on Twitter. Maybe it was just a coincidence, but whatever the reason was, it got me thinking about the role of risk intelligence in fashion, and in creativity more generally.

As my colleague at the American University of Beirut, Arne Dietrich, recently explained to me, creativity can be thought of as a product of two mental processes; one that generates new ideas, and another that evaluates these ideas. It is in the second of these processes that risk intelligence has a role to play.

Suppose a fashion designer is playing around with ideas for the next season. As new images flash across her imagination, she covers the pages of her sketch pad with simple line drawings. Then she puts down her pencil and reviews the designs she has generated. When evaluating these patterns, she will consider a range of criteria, some conscious and some implicit; her own aesthetic preferences, current trends in the fashion industry, technical aspects of production, and the likelihood that a given design will be unusual enough to get noticed and yet not so weird that it will never be worn.

Estimating that likelihood requires risk intelligence. A designer with low risk intelligence may overestimate the chance that a particular pattern will be a hit, or underestimate the probability that  a given design will catch on. A designer with high risk intelligence, however, will tend to get it just right, regularly providing realistic estimates of each new piece’s chances of becoming the must-have item of the season.

Good fashion designers construct mental models of what makes a good design slowly, often unconsciously, as they gradually accumulate experience.  These models may involve many different variables – the shape of the hemline, the way the folds fall, the texture of the material, and so on.  People with high risk intelligence manage to keep track of all these variables in their heads, but the process is unconscious; they need not be mathematical wizards, since most of the cogitation goes on below the level of awareness.  It is here that the difference between an average designer and an Alexander McQueen lies. Anyone can brainstorm a diverse range of crazy patterns; only a skilled designer can reliably pick out the ones that will sell.

50% = I have no idea

Benjamin’s recent post about differences risk intelligence between users of various web browsers drew a lot interest after it was featured on Slashdot. Among the many emails we received, one in particular caught my attention, because it articulated very clearly a common reaction to the whole idea of risk intelligence.

The email noted that our risk intelligence test presents users with a scale with one end marked 0% (false) and the other marked 100 (%true), and objected that this implied there was no option for “don’t know.” The email went on to reason (correctly) that “therefore the only logical choice that can be made in the case of not knowing the answer is 50%.”

So, what’s the problem? The instructions for the test state clearly that if you have no idea at all whether a statement is true or false, you should click on the button marked 50%. So why did the author of this email state that there was no option for “don’t know”?

I think the problem may lie in the fact that, while 50% does indeed mean “I have no idea whether this statement is true or false,” it does not necessarily mean “I have no information.” There are in fact two reasons why you could estimate that a statement had 50% chance of being true:

1. You have absolutely no information that could help you evaluate the probability that this statement is true; OR

2. You have some information, but it is evenly balanced between supporting and undermining the statement

So, maybe that’s what the email was getting at. But even if this interpretation is correct, it doesn’t justify the claim that there is no option for “don’t know.” There is. It’s the 50% option. That’s what 50% means in this context.

The email went on to add: “It’s very curious that you use a scale; surely someone either believes that they know the correct answer or they don’t know the correct answer. I can’t see that there is any point in using a scale. I would think it far more sensible to present three options of True, False or Pass.”

But this simply begs the question. As I pointed out in a previous post, one of the most  revolutionary aspects of risk intelligence is that it challenges the widespread tendency to think of proof, knowledge, belief, and predictions in binary terms; either you prove/know/believe/predict something or you dont, and there are no shades of gray in between. I call this “the all-or-nothing fallacy,” and I regard it as one of the most stupid and pernicious obstacles to clear thinking.

Why should proof, or knowledge, or belief require absolute certainty? Why should predictions have to be categorical, rather than probabilistic? Surely, if we adopt such an impossibly high standard, we would have to conclude that we can’t prove or know anything at all, except perhaps the truths of pure mathematics. Nor could we be said to believe anything unless we are fundamentalists, or predict anything unless we are clairvoyant. The all-or-nothing fallacy renders notions such as proof, belief, and knowledge unusable for everyday purposes.

In 1690, the English philosopher John Locke noted that “in the greatest part of our concernments, [God] has afforded us only the twilight, as I may so say, of probability.” Yet, as emails like this show, we are still remarkably ill equipped to operate in this twilight zone.

Internet Explorer users have low Risk Intelligence (RQ)

A hoax report earlier this year claimed that people who used Internet Explorer had a lower IQ than those using other browsers. Inspired by this bit of fun, Projection Point decided to carry out a poll to compare the risk intelligence (RQ) of people using different browsers. We found that Internet Explorer users performed worse than everyone else; they had lower RQ scores and were grossly overconfident.

We define Risk Intelligence as the ability to estimate probabilities accurately. Our Basic RQ Test consists of fifty statements—some true, some false—and your task is to say how likely you think it is that each statement is true. It’s a simple process; if you are absolutely sure that a statement is true, you assign a probability of 100 percent to it. If you are convinced that a statement is false, you should assign it a probability of 0 percent. If you have no idea at all whether it is true or false, you should rate it as 50 percent probable. If you are fairly sure that it is true but you aren’t completely sure, you would give it 60 percent, 70 percent, 80 percent, or 90 percent, depending on how sure you are. Conversely, if you are reasonably confident that it is false but you aren’t completely sure, you would give it 40 percent, 30 percent, 20 percent, or 10 percent.

When you have estimated the likelihood of all fifty statements in the test, the website will calculate your risk intelligence quotient, or RQ, a number between 0 and 100. Although our small sample size of 351 participants does not permit strong conclusions, they do suggest an interesting possibility; users of monopoly software (that historically has been responsible for many of the most severe software vulnerabilities) are not as good at estimating probabilities as their more adventurous counterparts. Perhaps the use of Microsoft Internet Explorer should be considered an indicator of poor risk intelligence. This would be consistent with studies showing that the computers of Internet Explorer users contain more malicious software than the machines of those using other browsers, that about 7% of downloads by Internet Explorer users are malicious and that the browser is amongst the most popular means of infecting Windows machines (this holds especially true for older versions). Although Microsoft’s efforts are slowly changing vulnerability trends for the better, these findings should come as no surprise given the company’s attention to security in the past: “Many of the products we designed [...] have been less secure than they could have been because we were designing with features in mind rather than security. [...] In the past we sold new applications on the strength of new features, most of which people didn’t use.” – Chief Research and Strategy Officer at Microsoft, Craig Mundie (2002).

Right now it looks like Apple users are the best when it comes to dealing with risk, a skill that should come in quite handy considering that Mac OS X was the first system to go down during the Pwn2Own hacking contest of 2011. But only time, a larger sample size and careful scrutiny may validate our observations.

Results

The test can be found at: http://www.projectionpoint.com/
A mobile version of the test for Android and iPhones can be found here.

Baseball, sabermetrics and risk intelligence

I’ve just been to see Moneyball, a new film based on the eponymous 2003 book by Michael Lewis. It tells the story of how Billy Beane, the general manager of the Oakland Athletics, led the team to a series of 20 consecutive wins in the 2002 baseball season, an American league record. This feat was apparently due to Beane’s use of sabermetricsMoneyball Poster

Sabermetrics is the application of statistical techniques to determining the value of baseball players. The term is derived from the acronym SABR, which stands for the Society for American Baseball Research. It was coined by Bill James, who began developing the approach while doing night shifts as a security guard at the Stokely Van Camp pork and beans cannery in the 1970s.

The drama revolves around the tension between Beane and the team’s scouts, who are first dismissive of, and then hostile towards, his statistical approach.  Rather than relying on the scouts’ experience and intuition, Beane selects players based almost exclusively on their on base percentage (OBP). By finding players with a high OBP but characteristics that lead scouts to dismiss them, Beane assembles a team of undervalued players with far more potential than the Athletics’ poor finances led people to expect.

There’s something very satisfying about seeing the scouts’ boastful claims about their expertise being undermined by newcomers with a more evidence-based approach. The same thing is occurring in other fields too, such as wine-tasting. In the 1980s, the economist Orley Ashenfelter found that he could predict the price of Bordeaux wine vintages with a model containing just three variables: the average temperature over the growing season, the amount of rain during harvest-time, and the amount of winter rain. This did not go down well with the professional wine tasters who made a fine living by trading on their expert opinions. All of a sudden, Ashenfelter’s equation threatened to make them obsolete, just as sabermetrics did with the old-fashioned scouts.

It would be wrong to conclude, however, that we can do away with intuition altogether. For one thing, you need lots of data and time to build reliable statistical models, and in the absence of these resources you have to fall back on intuition. If you have low risk intelligence, you’ll be screwed.

Secondly, risk intelligence is required even when sophisticated models and supercrunching computers are in plentiful supply. An overreliance on computer models can drown out serious thinking about the big questions, such as why the financial system nearly collapsed in 2007–2008 and how a repeat can be avoided. According to the economist Robert Shiller, the accumulation of huge data sets in the 1990s led economists to believe that “finance had become scientific.” Conventional ideas about investing and financial markets—and about their vulnerabilities—seemed out of date to the new empiricists, says Shiller, who worries that academic departments are “creating idiot savants, who get a sense of authority from work that contains lots of data.” To have seen the financial crisis coming, he argues, it would have been better to “go back to old-fashioned readings of history, studying institutions and laws. We should have talked to grandpa.”

Risk management is a complex process that requires both technical solutions and human skill.  Mathematical models and computer algorithms are vital, but such technical solutions can be useless or even dangerous in the hands of those with low risk intelligence.