Episode #203

Maria Franzoni (00:17): Hello. And welcome back to The Speaker Show with me, your host Maria Franzoni. In today’s show, we will be talking about judgment, bias and decision making. But before we get started, let me tell you that The Speaker Show is brought to you by Speakers Associates, the global speaker bureau for the world’s most successful organizations, providing keynote speakers for events, conferences, and summits. My guest today is a professor, writer and advisor, specializing in the quality of strategic thinking and the design of decision processes. The French strategy specialist has dedicated his career to coming up with practical tools and solutions to help people make the best decisions possible, which he communicates through his books, lectures and academic papers. He teaches strategy to decision making and problem-solving at HEC Paris and is also an Associate Fellow of Said Business School at Oxford University. Please welcome my guest today. Olivier Sibony. Olivier, thank you so much for joining me. How are you today?

Olivier Sibony (01:18): I’m fine. Thank you for having me, Maria.

Maria Franzoni (01:20): And you are most welcome. And I love your accent. I’m going to enjoy this so much. So, Olivier, I have to ask you, first of all, what got you interested in judgment and specifically cognitive bias? It seems to be a very specific area.

Olivier Sibony (01:33): It’s actually not specific at all. It’s some, it’s an activity that we all undertake every day. It’s something that we all have in common. We all make judgments all the time. What actually got me interested, to answer your question more directly is that I used to be a management consultant. I spent 25 years as a management consultant working with McKinsey. And I distinctly recall that even in my, especially in my very first engagement as a consultant, I was quite surprised to see that sometimes our clients who were brilliant people, they were the CEOs of great companies and they were working very hard and they were highly qualified and they were very competent and they were surrounded with great advisors like us, presumably, and sometimes they would make very bizarre decisions. They would make decisions that I would look at, even from my vantage point as the kid, really. And I would think this is really odd. You know, these guys are making a very bizarre decision. They seem to be making a mistake and my colleagues would agree with me. They would say, yep. They seem to be making a mistake. And so the question I had was why do these very smart people who should know better and who are trying very hard, sometimes make very bizarre decisions? That’s the mystery of judgment and decision making that I’m now studying for a living.

Maria Franzoni (03:01): And it’s fascinating. I’ve made some bad decisions in my life. So I’d like to. I’d like to understand a little bit more as well, actually as to why you might make a bad or strange decision. Tell me a little bit more though. You mention, we know we mentioned cognitive biases and that’s one of the areas, what is a cognitive bias for those of us who don’t know?

Olivier Sibony (03:18): So the first major explanation for those bizarre decisions that I was talking about is that we think in a way that is slightly different from the way we’ve been told in school or in our economics classes that we think. We are supposed to be, we’re told that we rational people who weigh the pros and cons of their decisions and who optimize for whatever it is that they’re seeking. And in fact, that is not how we think. We think using some shortcuts usually called heuristics. So we take shortcuts to get to the answer. And usually that’s fine. We use what, Danny Kahneman calls our system one, which is the fast thinking. Usually that’s fine. And that gets us to the right answer. But in some situations, our shortcuts are heuristics, leaders to make some systematic errors called biases.

Olivier Sibony (04:13): So the very simple example would be that if you are making a forecast, you tend to be over optimistic. You’re not going to be over optimistic every time, but you know, there’s a pretty good chance that you are going to err on the side of being too optimistic. That’s a type of error that is predictable. That’s a bias, it’s an error in a given direction and in strategic decisions in HR decisions, in all kinds of business decision and judgments, we see a lot of biases. We see a lot of predictable mistakes. So that’s the topic of some of my books. And that’s the work that I’ve been doing with a lot of companies to help them understand the biases that affect some of their decisions and understand of course, how to reduce those errors, not to reduce the biases themselves, that’s very hard. But to reduce the errors that the biases lead them to, by changing the way they think about their decision processes.

Maria Franzoni (05:09): That’s really quite concerning in a way because those biases affect every decision we make is that right?

Olivier Sibony (05:16): They absolutely affect every decision we make. It’s actually not that concerning. The good news is that the heuristics, the shortcuts that we take are usually fine. I was giving the example of optimism. Being optimistic is actually quite good airing on this side of overestimating ourselves. And thinking there are chances of success are pretty good, is much better than the opposite. You know, it beats the hell of being, you know, depressed and pessimistic about everything. So generally speaking, it’s not a bad thing, but of course, if you are making a big strategic decision on behalf of an organization on behalf of a company that is not necessarily your company, but by the way, it’s the same. If it’s your company, you would very much prefer to have an objective realistic assessment of the risks that you’re taking. That doesn’t mean you won’t take any risks, but you want to know what risks you’re getting into.

Olivier Sibony (06:11): And you want to make sure that the rewards that reward the risks, that you’re our commensurate with them. So in some situations it’s going to be quite concerning. And my, the focus of my research has been on those situations specifically in strategic decisions. So there are types of strategic decisions, types of big decisions that corporations take, that tends to be biased in a particular way. One prime example, for example, is acquisitions. We know from a mass of research, that companies that make acquisitions tend to be overly optimistic about their acquisitions. They tend to overestimate the synergies that they’re going to capture. They tend to underestimate the difficulty of capturing the synergies and the time is going to take, they tend to put on rose tainted glasses. When they look at the, at the target that they’re buying and quite reliably, we find that they’re disappointed and that in fact, the acquirers destroy value for their shareholders when they make acquisitions, while the sellers in fact do capture most of the value that is created. So that’s a pattern when there are patterns of errors like this, that always seem to go in the same direction. And I say, always, of course, it’s an exaggeration. But statistically overwhelmingly go in the same direction. That seems to be the sort of error that is attributable to a combination of biases. And there are quite a few layers.

Maria Franzoni (07:41): Wow. That’s really useful to know as well. Isn’t it from a decision makers point of view, to understand that those biases and understand what’s happening. That must be incredibly useful. Tell me a little bit about, because you’ve written a whole book about this, and we’ll talk about the book a bit later. You talk about noise being different from bias. What do you mean by noise?

Olivier Sibony (07:59): So here’s the, this is the new news in a way. We, you know, we’ve known about bias for a long time. And, lots of people, including myself, have been studying the effect of bias on big decisions, on big corporate decisions. Like the strategy decisions that I was talking about. Now, when we have analyze bias, when we’re done with bias, when we’ve sold for bias, if you will, are we perfect? No, we’re not. Unfortunately that would be good news, but unfortunately there is another resource of error and another component of error in strategic decisions or in any kind of decision. And that’s the random element of error that is left when you’ve accounted for bias. Now, this seems a bit abstract. So let me try to bring it to life with a very simple example from the realm of measurement, not of judgment, because it’s easier to figure if you think of measurement, suppose that you have a bathroom scale in your bathroom, and you’re weighing yourself every morning.

Olivier Sibony (08:58): And I suspect you are weighing yourself every morning as I am. And you, we all need to watch what the bathroom scale tells us in the morning. Now your bathroom scale may be biased in a way, for instance, it may be kind, it may underestimate your weights by say a pound. On average, if you compare the weight that it gives you to your accurate weight, weight, you know, measured by a very precise instrument, you might find that the scale is not perfectly accurate and that on average, it, you know, it gives you a gift of a pound. But that’s not all, if you step on your bathroom scale two or three times in quick succession on the same morning or in the same evening, you’re going to find that the reading is not exactly the same. Maybe it’s because you’re not stepping exactly in the same place, or maybe you are stepping on the scale in slightly more brusque way, or you’re for some reason, which you can’t quite figure out.

Olivier Sibony (09:56): It’s not going to give you exactly the same reading. That variability is noise. This, the average of the two or three measurements that you’re going to take that morning is going to be off by a certain amount. That’s the bias, but the variability around that average is noise. And if you only weight yourself once on that scale, the error between the difference between your true weight and the way that you’re reading is composed both of the average error and of the noisy error in this particular reading. So in every measurements that we’re taking, there is an error that is the average error of the instrument. And there is also an error that has to do with the variability, with the unreliability of the instrument. Our judgment works in exactly the same way when we are making a judgment, our mind functions, exactly like your bathroom scale.

Olivier Sibony (10:52): It has an average error, which is its bias. So on average, I will tend to be too optimistic when I’m looking at the synergies that I’m hoping to capture in acquisition. But if I look at three different projects, I will not be optimistic to the same degree in exactly the same way. There will be variability in my degree of optimism. And even if I look at the same project on several days, I might actually form a slightly different opinion because there are days when I feel more bullish about this project. And there are days when I feel more conservative. There are days when I’ve just heard someone speak very positively about the target. And there are days when I’ve just heard some people warn me about the dangers of this project. So I’m going to have an average error, which is a bias, and I’m going to have a random error around that average, which is the variability of my errors. And that is noise. And what we found in working on this book with Danny Kahneman and Cass Sunstein is that noise is very often neglected. We don’t talk about it much, but it actually is a very large source of error in a lot of very important decisions. It is very consequential. It’s also fairly easy to address once you become aware of it. And therefore it’s well worth thinking about.

Maria Franzoni (12:10): Wow, that’s fascinating. I knew nothing about noise, until actually, you know, I realized we were going to meet, we’re going to talk. That’s absolutely fascinating. I do know a lot about my scales being variable, and I tend, I don’t take the average Olivier I take the lowest.

Olivier Sibony (12:26): Well, that’s very tempting, isn’t it?

Maria Franzoni (12:30): Especially. Yeah.

Olivier Sibony (12:31): That would be a bias by the way.

Maria Franzoni (12:33): Yes, I know I am biased, but you know, I’ve been in lockdown for a while and the weight has gone on, but anyway, we digress, we digress. So what made me very happy when you say was the all statement at the end where you say that it’s easy to address, but before you tell me how we address it, are there different types of noise or is it that one type of the variable type of noise?

Olivier Sibony (12:54): There are actually several types. And it’s important to understand the difference between them. The simplest way to think about them is to imagine that you are, for instance, looking at judges in a courtroom who are looking at a number of defendants. Now you are the defendant today. You’ve been accused of some crime, and you are told that you are assigned to the courtroom of Judge Olivier, and your lawyer says, oh my God, Maria, that is really bad news. Olivier is a really tough guy. He’s a really tough sob. I really wish you had been assigned to the courtroom of Judge Smith, who is really a kind gentle, you know, kindhearted judge. But on average, we know that Olivier is much more severe than Judge Smith. So that’s the first type of noise. It’s the average level of the judgements of different people.

Olivier Sibony (13:54): And when you’re in system, like in this example, the judiciary system, where there is a random assignment to a particular individual, to a particular judge in this case, the choice, the luxury of who you happen to be in front of is going to make a big difference. It would be the same, by the way, if you went to a hospital and you asked to see a doctor about some particular type of disease. Some doctors will tend to over-diagnose a particular disease and others will tend to under diagnose the particular disease. Or if you were a student and your essay is going to be graded by your professor, all students will tell you that some professors are tough graders and others are kind generous graders. So there are differences in the average level of the judgements of different people, this is fairly intuitive. It’s fairly easy to see.

Olivier Sibony (14:44): It’s also fairly easy to correct if you think about it. So for instance, the university where I teach would tell us, you know, we know that you would have very different levels in your average grades. Now we ask you to abide by the following distribution so that this difference in level does not affect the grades of the participants, because there is no reason for that to be the case. So that’s the first type of noise. Let’s call it level noise, because is the difference in the levels of different judges? Then there is a second type, which I was talking about earlier, when I was telling you, if I’m in a better mood, I might actually think more highly of the synergies. And if I’m in a bad mood, I might think more defensively about the synergies. This is actually what we call occasion noise.

Olivier Sibony (15:30): It’s the difference between judgements of the same case by the same person on different occasions. If we go back to the example of the judges in the courtroom, we have evidence that judges will be more severe on the day after their favorite football team has lost a big game. So you, we are talking today and France lost to Switzerland in the European cup last night. You don’t want to be facing a judge in France today. That’s a very bad idea. We also know that judges are tougher before lunch than they are after lunch. We know that doctors will prescribe more opioids in the afternoon than they do in the morning, but they will presscribe fewer cancer screening tests in the afternoon than they do in the morning. So a lot of factors that should not make a difference that purely have to do with the occasion.

Olivier Sibony (16:23): They have nothing to do with the case and very little to do with the individual making the judgment. Factors that should not have an influence do in fact, have an influence, extraneous factors make a significant impact, not a very large impact in general, but a significant impact on decisions from one occasion to the next there is going to be variability that’s occasion noise. So we’ve talked about differences between judges. That’s the difference in levels. And we’ve talked about differences within the judges. That’s the effect of the occasion. And we think that’s it, right? We think, you know, what could be left when we’ve covered these two, in fact, there is a third source of noise. It’s a bit harder to wrap your mind around it, but it’s the largest and the most important one and the most interesting one. And here’s what it is.

Olivier Sibony (17:13): If you’re looking at your judges in the courtroom, Judge Olivier and Judge Smith. Olivier may be tougher than Smith on average, but maybe, just maybe he’s going to be kinder on you because he likes you, or because he doesn’t think that white collar offenders should be punished so severely as the average punishment they get in the system. So Olivier and Smith both have their tastes, their preferences, their idiosyncrasies, which are going to make the ranking of their judgements difference. On average, Olivier will be more severe than Smith. But if you ask both of these judges to rank all the cases from the most severe one, from the one that should get the worst punishment to the one that is the least offensive, their ranking is not going to be identical. They have different tastes. They have different preferences because they have different histories.

Olivier Sibony (18:17): They have different personalities. They have different political views. They have different biases, basically. They are different people. And when we’re asking different people to make good judgment, looking at the same facts and being in the same mood, leaving aside the fact that they are variable from one moment to the next, they are going to make different judgements simply because they are different people. That is by far the larger source of noise. We call it pattern noise, because the pattern of the judgements of our two judges in my example are going to be different. And it simply reflects our personalities, our uniqueness, the fact that we are all different, which in general is a very good thing and makes life very interesting. But if you’re expecting judgements to be identical, cannot be a very good thing.

Maria Franzoni (19:07): That is incredible. That is really fascinating. And as you are going through those various examples, I’m thinking of times in my life, when I’ve either had a decision made for me, or I’ve made decisions. And I’m thinking, actually, I recognize what you are describing. I’ve not been in court by the way, but, I’m recognizing the thing yet. There’s time. There’s time.

Olivier Sibony (19:27): Just stay away from it if you can.

Maria Franzoni (19:29): Yeah. And having mentioned the fact that France lost, you know, the day before we are recording, it’s very good of you to be so kind today, to me, being a Frenchman. So that it’s, that is fascinating. And obviously you talk about all of this, and you mentioned the book, let, let’s just give people the name of the book. It’s called Noise: A flaw in Human Judgment. And you’ve co-written it with Danny Kahneman and Cass Sunstein. And it’s a Sunday Times best seller here in the UK, by the way. I don’t no if you knew that.

Olivier Sibony (19:58): Yes. I knew that and it’s good to know, in fact.

Maria Franzoni (20:01): It’s good to reshare it. So that’s fantastic. So why are we discovering noise now?

Olivier Sibony (20:10): It’s, itself, this is a very interesting phenomenon. Why are we paying so little attention to knowledge? Why, you know, since it’s such a large source of error, why aren’t we addressing it? Why aren’t we talking about it? And I think the answer is twofold. You know, first as individuals, we don’t ask ourselves, what would someone else think of what I’m thinking? We just look at facts at cases and judgements that we’re making. And especially if we are professionals, if we are making a considered thoughtful professional judgment, we assume without giving it a second thought that another qualified professional looking at the same case would come up with roughly the same answer. We did an experiment in an insurance company, for instance, where we asked underwriters, the people who set the price, the premium that you’re going to pay for your insurance policy.

Olivier Sibony (21:12): We asked them to look at the same case, and we asked different underwriters separately to look at the same cases we compared, what prices they put on those policies. But before we did that, we asked them, how much do you expect the difference between two underwrite to be? And their answer was, well, of course, we expect some difference. We don’t expect to be perfectly identical. We’re not machines, we’re experts, we’re human beings, but you know, we’re applying rules and procedures and tools of the trade that are supposed to make our judgements consistent, uniform, homogenous. So we think that on average, we should defer by about 10%. That would be tolerable. It turns out that the difference between two randomly chosen underwriters is closer to 55% than to 10%. It’s more than five times larger than people suspect. And the reason is they never actually stop to ask themselves, what would the underwriter next door think of this case?

Olivier Sibony (22:13): They’re looking at the case. They’re applying their best thinking. They think that the world is as they see it, and that everybody would see it in the same way. They think that the case they’re looking at is what it is and that their colleagues would see it exactly the same way. And so they don’t actually realize that assumption, that assumption of agreement, that assumption that we’re making, that people are seeing the world in the same way that we’re seeing it, that assumption is actually wrong. That’s the first reason. Then there’s a second reason, which I think is actually more important, which is that organizations do not do their job very well. Organizations value harmony, they value consensus. They value maintaining the illusion of agreements more than the value, the quality of their decisions. So the underwriters in our example, had never been confronted to the experiments that I was talking about, which we call a noise audit.

Olivier Sibony (23:17): The insurance company had never actually tried to give two or three or five underwriters, the same case, and to compare what they independently come up with. It’s not very hard to do if the organization does it, wants to do it. But the reason it doesn’t do it is that it doesn’t want to expose the extent of noise. It’s kind of an embarrassment for that organization to realize that the experts that it relies upon in fact are so variable, are so noisy in their judgements. And rather than expose that problem and then have to deal with it. The tendency of many organizations is to sweep the problem under the rug and hope that nobody notices it.

Maria Franzoni (24:03): Yeah, I can imagine actually. So that makes sense to me that they don’t want to expose that. And it also makes sense when you’re explaining about the underwriters with that huge margin of difference. It makes me realize why my insurance policies can vary so much if I go forward to one company to another that’s incredibly exactly. And that 55% rings true actually with my recent car insurance. So that is really fast. It’s fascinating. I’m learning a lot about my life from you here. So what can we do about it then is the answer to have more people, you know, is it to think about what somebody else would say or have more people actually looking at each decision?

Olivier Sibony (24:40): It’s both, first of all, the answer is to do what I was talking about in the example of the insurance company. Do a noise audit. Realize what the actual magnitude of noise is. If you are not aware of the problem, it’s hard to motivate yourself to fix it. So, you know, expose the problem. This is not as easy as it seems, because again, it creates some friction, some tension. There’s a funny story that we heard from, a colleague who worked with universities and he was helping the admissions departments of universities to improve their admissions processes. And the university that he was working with had two graders, two admissions officers read each essay and separately grade the essays, except that the second rater would actually see the grade that had been assigned by the first grader. So our colleague told them as we would’ve told them, you know, that’s actually not a good idea. You want those grades to be independent. You want them to be given separately. So you should hide the grade of the first grader so that the second grader makes a completely independent uninfluenced judgment. And their answer was, oh yeah, of course. That’s how we used to do it. But we disagreed so much that we chose to move to the current system.

Maria Franzoni (26:02): Oh my goodness. Wow.

Olivier Sibony (26:05): You know, that’s touchingly transparent. But in fact it’s quite typical. It’s usually not that clear, not that overt, but the ways organizations design their procedures is usually intended to suppress disagreement rather than to expose it. So you don’t want to do that. You want to expose the disagreements. You want to make sure that the noise is out there, that you hear it, and that you become aware of the problem. That’s what the noise audit does for you. Then what you need to do to reduce the noise is put in place some practices, which we call decision hygiene and what we mean by decision hygiene. The reason we have this slightly offputting term and that’s intentional is to remind you that decision hygiene is a bit like washing your hands. It’s a good idea to wash your hands. It’s good practice.

Olivier Sibony (27:01): You should it. But if you’re successful, if all goes well, you will actually never know what particular disease you avoided by washing your hands. You are not washing your hands and thinking, Hey, here goes the germ for COVID 19. And here goes the virus for the flu. And here goes, you know, you won’t know what problems you’re avoiding. That’s what good prevention does for you. So decision hygiene is a bit like this. It’s putting in place processes to make your decisions that are going to make your decisions more robust, less exposed to noise without in fact, knowing precisely in what direction the error would’ve been. If you had made an error. How would you do that? As you were mentioning earlier, one way to do that would be to average the judgements of several people. So in our example of the admissions officers, having two or three admissions officers read the essays separately, and then taking the average of the grades that they’re giving would be a pretty good way to reduce noise. That’s guaranteed to reduce noise.

Olivier Sibony (28:04): It’s costly because you double or triple the amount of time that you spend on each decision, but it’s a pretty good way to reduce noise. Another very good way to reduce noise, which is less costly is to structure the decisions that you’re making. So to break down the decisions into components and to evaluate those components independently of each other. That’s a very good technique in recruiting decisions, for instance, where you would want to use a structured interview, where you define the questions that you ask and you evaluate the candidates separately on each of the questions. That’s another good technique. A third example of a way to reduce noise of a decision hygiene technique that works very well is to use guidelines, to create a simple set of rules that limit that cabin, the discretion of the judge that set a constraint around the discretion that an individual has, or at least that give that individual an indication. In medicine,

Olivier Sibony (29:06): There are lots of guidelines which are not intended to prevent doctors from making judgements, on the contrary, they’re intended to help doctors reach the best possible judgements. And they are in many cases, quite helpful, not in every case. It’s difficult to design a good guideline. It’s not possible for every condition, but for many things, guidelines make a very big difference and help improve the quality of judgment. There are more decision hygiene techniques, but these three averaging structuring and guidelines are actually quite practical and simple things that you can start thinking about for any type of decision that you’re making. And they’re pretty sure to reduce noise.

Maria Franzoni (29:48): I like that. That it is actually, it does sound quite simple when you explain it and it, I really like those three solutions. Thank you. And I appreciate that there are more, we kind to the end. So I wanted to come to your speaking because obviously this is a speaker bureau podcast, and you’re available for hire. When clients come to you and say, Olivier, come and, and speak to us, are they looking to you to help them identify noise? Or are they looking for the solutions? What can you help them with? Because it’s quite such a big topic.

Olivier Sibony (30:17): Well, whether it’s bias or noise or both really, because it’s judgment that I took a about in general, people are interested, I think first and foremost, in gaining insights into how their mind works and people find it fascinating. We usually play a few games and they can see from the results of those games, how their judgments, you know, differ or depart for on what should be the rational decisions. So they have, you know, an aha moment when they realize that their mind does not work in the way that they thought their mind worked. And that gets them to the point where they say, oh my God, I must be making a lot of mistakes. Exactly. As you were saying earlier, oh, that’s very concerning. And so we then typically get at, into the solutions, it’ll be a long story to cover all the solutions, but I always try to make a point to give people a very practical thing that they can do, you know, on the proverbial Monday morning, it doesn’t have to be a Monday and it doesn’t have to be a morning, but you know, something that they can do in practice so that they have something tangible that they leave with.

Olivier Sibony (31:24): So my aspiration, when I’m speaking is that people have a good time that they actually learn something about themselves and have an experience that is enjoyable, but also that they take away something practical and science based that is actually going to make a difference to the quality of their decisions.

Maria Franzoni (31:41): They cannot ask for anymore. That is perfect. Olivier, it has been a pleasure. I hope you enjoyed yourself.

Olivier Sibony (31:47): It was a great pleasure. Thank you very much, Maria. And good luck to Italy for the rest of the European cup.

Maria Franzoni (31:55): Okay. I don’t know when this is going out. We might know by then that Italy is won. Let’s find out. Did you see that? I said that they’ve won. I’m being optimistic cause it’s a forecast. Right? Anyway. Thank you so much. And thank you for listening to The Speaker Show. If you enjoy this episode, please leave a rating on iTunes and you can keep up with future episodes on the Speakers Associates websites. That’s at (speakersassociates.com) or on iTunes, Google Podcast, or on your favorite podcast app. And don’t forget to grab a copy of a Olivier Sibony’s latest book Noise: A Flaw in Human Judgment, co-authored with Daniel Kahneman and Cass Sunstein. So goodbye for me and see you next week. Bye bye.