Introduction

Patrick: [00:00:49] My guest this week is Annie Duke, and the topic of our discussion is how to improve decision-making. We break decisions down into their component parts, values, beliefs, decisions, randomness and outcomes. After diving into each, we discuss how to make better decisions, how to work in group settings and how to harness the power of tribes and identity to improve our behavior. Annie has thought about this topic as much as anyone and her methods for getting us to think in probabilities, and to stop evaluating decisions based on outcomes tainted by randomness will be useful for anyone listening. Please enjoy.

Annie: [00:01:19] Here's the main thing I think about, there's so much information out there, we know what's right to do. So we can look across lots and lots and lots of data and say for the average person who's investing, what's the best thing for them to do? And we know, don't be an active investor, put your money in, walk away and let it sit. And we know that, that's true, but what's interesting about that is that despite the fact that that information is readily available, that people know it, people don't do it. And so the question that I'm always trying to think about is why? And I think about it in two ways, one is, how much are we communicating with people in a way that activates identity and tribe in a really positive and good way, and then how are we thinking about what the rules of the game are?

So on the tribe side, I feel like tribe gets a very bad rap, not surprisingly, because we see it in the political context all the time that people were fighting with each other. So we get this idea that like activating identity is like really bad, really destructive. And obviously, you look at the political arena and you can see that sometimes that's true, but it's not always true. So if we think about back to why did we as human beings form tribes in the first place, obviously it had to be adaptive otherwise people who were in tribes wouldn't have survived as much as people who weren't in tribes. So we know that there were some selection for tribes, and why was that?

Well, human beings don't have big claws and we don't have big teeth and we're not really strong and we're terribly slow and we're kind of weaklings for our size, if you think about us compared to the lions and bears and tigers, Wizard of Oz type of thing. So what did tribes do? It's like we formed these groups that were generally based off kinship. So they were like these kinship groups and now we all work together because we had brains. So we could figure out how do we defend our territory if we all kind of work together.

Now, obviously part of working together like that is that now you, Patrick, are going to do something that's really hard that you might otherwise not be willing to do. So, for example, in defense of the tribe, you might fight a bear, which normally you would run away from a bear, but now you're protecting your kin, and so you'll actually fight the bear. You might march 100 miles in order to find a resource, for example. So the bad part of tribe comes from, okay, so we're going to protect our own little group and if another group tries to come and invade they are bad, they are other, whatever, and we're going to kill them. But the good part of tribe is that it can help you to do things that are really hard that you otherwise would not do, that would be against what generally your instincts might be to do. So how do you sort of get that working for you?

Well, we can think about what is tribe offering us, and there's really wonderful work by a guy named Jay Van Bavel, and I really highly recommend him as a follow on Twitter, I think he's a great science communicator. And he does work in tribe and identity and the intersection of identity and beliefs. And he kind of frames these as a few of the things that tribe is giving you. One is, what are the things that we know or don't know? So we could think about, we know we should hold, so tribe can help with that. But the other thing is distinctiveness and belongingness, and these are two incredibly important things for human beings. We want to feel like we belong to something, but then also we want to feel like we are distinct from other people.

So I really feel like when we're really thinking about it we know what the data is telling us in terms of the right thing to do. Instead of just talking to someone and being like, here's the data and this is what I think you should do to start to activate identity and say this is what we do and this is what we do that is distinct from other people, and when we think about the group of people that succeed and the group of people who fail, this is what the people who succeed do and I'm going to warn you, It's really hard. So it's like, kind of saying, you're going to have to fight a lion sometimes, it's incredibly hard. When the market drops 5% in a day, it's really hard not to care, to really say, "I'm not worried about this, this is not part of my strategy." It's really hard not to look at your balance sheet every month and see whether it's gone up and down, it's hard. Just like it's hard to not look at Twitter every three seconds. We want to see how we're measuring ourselves.

But if you say, "This is what we do as a group and this is what other people do." Other people are always checking that, they're always reacting to it. When they see that their balance sheet has gone up and they're looking at it and they're saying, "But I had these bond holdings and boy, I could have made so much more money if I didn't have my money in bonds, why wasn't all my money in stocks?" And you tell them in advance, they're going to feel like that, that's what other people do. When the market is down they say, "Why did I have any money in stocks? Why wasn't all my money in bonds?" But you have to know that you can't know that those things are going to occur until after the fact. So what we do is think about, given that we don't know how the future will turn out, we know statistically this is what the right thing to do is. But we also know it's really hard to stick to it, and that's what makes you special.

Rule Setting as a Superpower

Patrick: [00:06:33] Talk about how that idea of harnessing the positive powers of identity could play into rewriting rules of the game. So if that's sort of the second avenue in this idea that we were talking about, what are unique ways that people can think about rule setting as sort of decision making superpower?

Annie: [00:06:51] I think that rules are like an amazing way to go. So let me just give you an example of a difference between setting a rule and sort of trying to get to a goal versus having a rule. So let's say that my goal is I'd like to eat less meat. Every single time I'm confronted with meat, I have to make a new decision. I'm sitting in a restaurant, at the grocery store, whatever, I have to decide, it's a fresh decision. Do I want to get something with meat or do I want to not? But if I say I'm a vegan, now that's a rule, it's part of my identity. So therefore I know I'm no longer making brand new decisions every single time that I confront a menu or every time I confront the grocery store or whatever. So setting a rule, I'm a vegan, actually sort of saves me from the opportunity to make a lot of mistakes. But in order to kind of get to those rules...

So you could think about part of the identity of this tribe is that we're going to think about our identity and we're going to write it as a rule. For example, you could say, "I only look at my balance sheet once a year." That could be one thing that it is. It could be just, "I look at my balance sheet, but I don't act on it without talking to my advisor first." Or, "I look at my balance sheet and I recognize that when I'm sad, that's what other people do and so I don't react that way to it." Or, "I'm a long holder." All these different ways where instead of saying my goal is to hold for the longterm, which is kind of like saying my goal is to eat healthy, you say, "I'm a long holder." And now that becomes part of your identity, it's kind of a rule about the way that you behave.

So in order to do that though, in order to get to what is the rule, which really is kind of the rules of the game, like how do I win the vegan game? Well, obviously, I don't eat meat, that's what the vegan game is, so every single time I don't eat meat, I score points. You have to identify what your values are, that's kind of step one. So the first thing in sort of getting people to think about what are the rules for me is to actually sit down and identify their values with them, which I think we don't do enough of. I think that we assume that people generally have the same values, and I think that at the top level, that's true.

So I think at the top level, it's like, I want to provide for my family, I want to retire with enough money to live a good life, whatever it might be, I want to make sure that I'm covered in case I get sick, I want to have enough leisure time to have a good life, whatever it might be like. At the top level, this is kind of like, I want to find a job with meaning, I want my life to have meaning, I want to provide for my family, these kinds of things. Yes, at the top level, we all have the same values. But when you get down into the core of it, what you can see is how those values express for one person versus the other, like super duper different.

So for example, what does provide for your family mean? Does it mean money? Does it mean time? Are you somebody who's like, "I'll work 100 hours a week because I want to make enough money that my kids, I'm never going to have to say no to them for financial reasons." But obviously if you're doing that, you're sacrificing a lot of time with your children. Maybe you're someone who's like, "I don't want to be one of those 100 hour a week people, I want to make just enough money that my children have their basic financial needs met and we're fine, and they're fed and they have a house, but what I want to provide for them, what providing for my family means to me, it's time."

If you're like, "I want to live a meaningful life," what does a meaningful life mean? Do you get meaning from your work? Do you get meaning from your family? If you get meaning from your work, what is the meaning from the work? Is it I really enjoy intellectual work, I actually want to be physically feeding people who are in need, I want to stand and hand out food to people who are in need, I want to build houses for people who need houses, I want to physically be doing it. Or maybe for you, meaning is I'm an effective altruist and I think that I could make a lot of money, and so therefore it would be immoral for me to build a house for somebody, I need to be making money. And when I do that, I can then donate houses to people, and this is the way that I'm going to find my meaning, it's that way.

So you can see it's like a whole different thing and it gets down to as simple as if we both go into a restaurant and we're trying to order a good meal, which I assume is what we're all trying to do, what does good mean for you versus me? For me, maybe it's I want to order something healthy or vegan. For you, It might be I want to order the tastiest thing on the menu regardless of what the health profile of that is. So we can see that in small ways and big ways. So if you actually take the time to sit down and figure out what do these things mean for you, now, you know and you can work with them about what the rules of their game are.

Patrick: [00:11:12] What I'm taking from this and have this huge kind of visualization in my head and it's a good entree into the meat of our conversation today which is this over focus on outcomes in terms of evaluating the quality of a decision or just in general, things that happen probably sway us more than they should when we should instead be focused on the process of the decision. But what's interesting now is that we've got this underbelly of process which goes all the way down to the root of values. And so I want to spend a few more minutes there and get your take on how fixed those ideas are? How much can you choose your values versus them choosing you? So eating the cheeseburger versus being vegan, health versus tastes, how much of this do you think is under someone's control versus some sort of natural default settings in a person? Because I think that you're right, that these are incredibly powerful culture at affirm. Once it's set culture or values, these are things that are very enduring through time, which I think is the bedrock probably of good decision making, but I want to make sure you agree.

Annie: [00:12:04] First of all, I think it's true. I think that at the base, what you value, what the rules of the game are, for you, and what it is that you're trying to accomplish and really properly identifying that is really at the base of all good decisions. And so, as an example, you can think about how much are you focused on short-term wins versus long-term wins. So if I realize overall, my goal is to be a better decision maker, I can now think about a value of accuracy. I can think about, for me, one of the things that I really value is moving toward constructing an accurate model of the objective truth. That's actually quite a hard thing to do, but this is my value. So what comes out of that is one of the ways to win that game is to approach being wrong with excitement, that sort of something that falls from that.

Now, what's interesting about that is that it allows me to turn something that would be a short-term loss into a short-term win. Because normally when we find out that we're wrong about something, that's a loss, it's like a loss for identity. Now that turns into a win because I'm like, "Oh, yay, I'm wrong. Now my decision-making is going to be better." I can feel good about the way that I react to that and then obviously that's going to improve my decision making going forward. But the thing that you said, which I think is true is that temperamentally, people have different temperaments. So some people, I think naturally have more self control and some people naturally have less self-control, for example. I think some people are naturally more rational and some people are naturally a little bit more emotionally driven. And I think that those kinds of temperaments, we have to recognize that there's a whole range in terms of what's the quality of the decision for you, Patrick, your range of decisions, what's the quality of my range of decisions or somebody else's range of decisions?

See that's totally fine, what we're trying to accomplish is two things. Thing Number one is let's make it so that more of Patrick's decisions are out at the right side of the distribution for him. So you've got your worst decision and your best decision, let's try to make it so more of your decisions are sitting out at the right tail as possible. But then I think that through this really good decision hygiene, what happens is that just like anything else, those kinds of habits of mind start to get reinforced. And as those good habits of mind get reinforced, that actually increases the quality of your decisions in general and so you can actually shift the distribution to the right. Now, you can imagine one person, just temperamentally, might have a distribution that's further to the right than yours. But imagine if you just shift a little bit right, you might not get all the way there but think about for you, because what I care about is you, what's going to happen in terms of the quality of your decisions in the long run? Those really small changes really matter.

I feel like we get really focused on how do we get people to make perfect decisions, how do we get people to not make mistakes, and I think that's a crazy frame to be looking at that through. It should be, how do I get you to make fewer of them? Because just a little bit less is going to have a huge impact on your life. If you're somebody who is going to panic every single time, literally every single time that the market drops more than 3%, you're going to be panicking about that, what if I could get you to do that only 80% of the time? Now someone might view that as Ooh, what a failure, because you're still panicking 80% of the time the market drops over 3%, but you used to do it 100% of the time, that's a huge success for you. And if you think about what are your financial outcomes going to look like in the long run that I've now gotten you to panic less of the time, think about how huge an impact that's going to have. And that's all good. So I think we need to recognize that really, this is all about how are we moving people because you're born with the brain that you have, I can't take it out of your head and stick a new brain in there. What I can do is start to reinforce better habits.

Anatomy of a Decision and Resulting

Patrick: [00:15:51] Let's use that as an excuse to talk in more detail about the specific anatomy of a given decision. So obviously luck plays a role in all of this, and we're going to talk a lot about that, and that's a key part of what then determines outcomes, which we already mentioned is the mistake is focusing on the post luck outcomes. But when it comes to a decision itself, sort of the atomic platonic decision, talk us through the inputs or the pieces of that that matter most, and then we'll talk about ways that hopefully we can improve on that for people.

Annie: [00:16:17] Do you want to know something amazing? No one's ever asked me that in a podcast. So I really appreciate that. So we can kind of think about, as you said, I like to think about the anatomy of a decision. So at the base of the decision is your beliefs, this is stuff you know and opinions you have, basically what are your beliefs about the world? Now, I say stuff you know but let's just be clear that it's rare that you know something 100%. The example I give is I can't be 100% sure that the earth is round because I might be living in a simulation, there always should be a tiny bit of doubt. But obviously there's a range of how sure are you of the things that you believe. So beliefs are really the stuff you know, and then there's also stuff you don't know. Okay, so obviously stuff you know is going to be informing the decision, but remember that stuff you don't know has an influenced on your decision as well because they're in that box of stuff you don't know. Some of that, If you knew it, would be really helpful for the decision.

So the most important thing that's sitting in the stuff you don't know box is what's the actual outcome that will occur, and you can never know that, ever. So we don't want to think, well, we shouldn't think about stuff we don't know because that's not actually an input into the decision because it is in sort of the omission. So we've got those two boxes, stuff you know, stuff you don't know, those are obviously informing the decision itself. You make the decision, that now gets you to an outcome, and this is where we get tripped up. It gets you to a particular outcome but that's a temporal problem because at the moment that you're making the decision, the minute you make the decision, between the decision and the outcome that actually occurs, there's an influence of luck. Because when you make a decision, there's always lots and lots and lots of ways that the future could unfold.

So there's always multiple possible outcomes, but there's only one that will actually occur. And that's where we really get into problems in the next part in terms of the anatomy of the decision. Because once you have the outcome, that's now your experience. The experiences of your life are all because there was one particular path that happened and other paths didn't, but they could have. So now we take those paths that happened and post that we try to figure out, what did we learn? What did we know before? What do we know now that we didn't know before? Was the decision good or bad? Were there things that I could have known that I didn't know? And you're sort of trying to walk through all of these decisions, hopefully with the goal of now making a better decision going forward. Sadly, it's mostly with the goal of ruminating that you actually made a not so great decision before.

Just again, to review, you have stuff you know, stuff you don't know, those that inform the decision, you make a decision, that decision now determines the set of possible outcomes that can occur and the chances that any one of those outcomes occur. So as we're weighing options, that's really what we're doing. Option A has some set of possible outcomes, option B has some set of possible outcomes. They could be the exact same set, but different probabilities, they could be a different set that could be overlapping, it could be a subset, all those things and then you have some chance of those occurring. And then post decision, there's an intervention of luck, that's anything you can't control, and then you have the actual thing that occurred. And now under the influence of the actual thing that occurred, you're supposed to try to figure out something about what the decision quality was.

Patrick: [00:19:27] I want to make sure we touch on the good things that people do in this process and the bad things, could you talk about your concept of resulting, what that means?

Annie: [00:19:34] So here's one of the main ways that trying to do sort of a post outcome review, which is mostly what we do. Like that's not just something that happens in business, this just happens in our minds all the time. One of the big problems with the post outcome review is that if you think about it, particularly as you try to think about, so how do we make a decision? This is really hard because we have the stuff we know, we have the stuff we don't know. And then within the stuff we don't know, there's stuff we can't know, the size of the universe or what you're thinking right now, things that have to do with predictions like I don't know what the weather will be like on this day, next year exactly, I have a range but I don't know, It depends on where I am in the world. So there's all sorts of stuff that we can't know.

But then there's stuff that we could know, but maybe we don't know right now, but then we have to figure out, well, how much does that cost? So that's hard. Some information is too costly either because of time, it would take too long to find it out, and so then you lose all your opportunities or maybe in terms of money, how much is it going to cost you money wise or resource wise in order to get that information in terms of the return that you'll get in improving the decision that you make. Then within the stuff that you could know, there's stuff that costs too much and stuff that is sort of at the right cost. And then there's the problem of, very often you don't know what you don't know or you don't know how much it will cost. So this is all very complicated.

All of that very complicated stuff is informing the decision you make, which now has this further complication of the decision you make doesn't guarantee a particular outcome, it guarantees a set of possible outcomes that are probabilistic in nature. Those probabilities are rarely exactly known, they are for some things like the divorce rate, but they aren't for most things like whether a stock is going to go up or down. And then it's also really not known as we're trying to figure out what the possible outcomes are, there's going to be lots of things that happen that we didn't actually see on the horizon. So it's hard to figure out what all those possible outcomes are, so that's all like super hard. So given that, that's all really hard, as we look back on the decision, the only way that we can figure out was it a good decision or was it a bad decision is to reconstruct all of that stuff.

A lot of which is hard in the moment, but now looking back on it, trying to figure out, well, what did you know, what didn't you know, what could you have known, was it the right cost, did you have all the possible outcomes figured out properly, what were the percentage of those outcomes? Oh my God, everybody's head exploded. I need like a super computer in order to figure that out. So that's hard, it's not transparent, we can't see the math, we don't live in the matrix where like all that code is coming down for us so we can look and say, "Oh yeah, no, look, there was a bug there." Who knows? Like that's too hard, it makes my head explode.

So what is the thing that I do know? I know how it turned out, that is sitting right in front of me. And whatever my values are, I know that for me, whether that was good or bad. So I know that. And so what happens is that our brains go, "Okay, that's stuff tough, but this, I know it turned out well, it turned out poorly. And so I'll use that as a shortcut to get me back to whether the decision itself was good or bad." So this is a general problem that we have that has to do with over fitting outcomes to decision quality, thinking that outcomes are much more predictable than they actually are, given the decision. This is called resulting, and you can see why. I take the result, I know whether the result was good or bad, and that tells me whether the decision was good or bad. So that's why it's called resulting, it's an intuitive name, let's put it that way.

In the book, I talk about a really very famous example of resulting, which was Pete Carroll in the Super Bowl. And this actually is a good way to see how hard getting back down into the decision quality is. It was second down, 26 seconds left, down by four to the Patriots because it's only the Superbowl if the Patriots are in it. So they're down by four so obviously they have to score a touchdown, it happens that Pete Carroll has one time out. He does something super duper unexpected, he calls for a pass play when everybody's expecting him to hand the ball off to Marshawn Lynch, who's the running back, the pass is intercepted. And universally, everybody does this resulting thing, they're like an interception is terrible, I can see what the quality of that was. So they all say this is the stupidest decision ever.

So if we go back and we think, well, what did he know? What didn't he know? What were the different possible outcomes? What were the percentage of those outcomes? Because we know that that's the stuff you need to know in order to really know whether it was a good decision. There's a lot of other outcomes in these, but the three main outcomes of the pass that we would care about would be touchdown, incomplete and interception. And obviously there's things like fumble and sack and whatever, but those are kind of the same either way, whether you hand it off to Marshawn Lynch or not, so let's just ignore those for a second to simplify. So here's what we know, the only way, literally the only way that we know whether this is a good decision or not is to understand how often the outcome which is interception, occurs. And if we don't know that, we don't really know a lot about what the decision quality is.

The thing is that if I go around and I ask people, they're like, I don't know, I don't have like all that, I didn't even think about that, I didn't know that was the thing I should be thinking about, it's really obvious that it was a bad decision because it ended in an interception. So if you ask yourself that question, the answer there is between one and 2%. It starts to become pretty clear that, "Oh, wait a minute, maybe that wasn't such a bad decision after all." So you can walk through it, obviously a touchdown, we know what happens, he's a genius, which is also resulting by the way. So that's just as bad if just because a touchdown is scored, they say he's a genius.

And I think if everybody takes a second to think, "Okay, what are the headlines look like if it's a touchdown?" I think everybody can feel like, "Yeah Belichick Belichick." Like, this is why he's so great, he stole the fifth Superbowl from the jaws of defeat. Oh, look, Tom Brady who's sitting on the field crying again, so sad. So, that obviously, he's a genius.

Now what's interesting is in the case that the ball is incomplete, a really important thing happens, which is the clock stops and it stops really fast. So we know that, that's actually not a bad outcome for them because the clock stops fast enough for them to still get two plays, so that's kind of a really good thing for them. So it's like, yeah, do they not score a touchdown? No, but it's in the world of good outcomes for them, and then you have this like one to 2% chance that the ball gets intercepted. So just on that basis alone, you can see what the power of resulting does. It seems so obvious that nobody even asked that question. Like I challenged people to go... Don't look at Nate Silver's site because you'll see some pretty good math there, but if you look at like USA Today and the Washington Post and whatever, look at that Monday what people were saying as they were declaring that Pete Carroll was an idiot and you will not find that statistic in there anywhere. Why? Because people are like, "Well, it's obvious, why would I need to know that?" But that's exactly the thing you need to know. But the problem is it's so complicated, so this is a way to simplify. So resulting is basically a simplification, but this isn't a place that you'll actually... It's not a good place for simplifying.

"Want to Bet?"

Patrick: [00:26:26] One of my favorite things about your work is these simplifying little heuristics and questions that you can ask. And obviously the famous one that everyone has glommed onto, rightfully so is, "want to bet?" Because it forces you to think about the various probabilities and then base your decision based on expected outcome versus black and white type thinking. So range of probabilities instead of just being a deterministic person about it. The other one that I found really interesting in there is this question of, how do I know if the base level of the anatomy of a decision is beliefs, which is like the information you think you have about the world? I'd love you to expand on that question because it's amazing. If you start to turn that lens on yourself where you've got a belief and you start to ask the question, how do I know? Things can fall apart pretty quickly. What else is behind that question of being skeptical about your own beliefs?

Annie: [00:27:09] Well, first of all, I think at the base of that is this idea of when you say to somebody, do you want to bet? I think there's two things that are coming into focus. Thing number one is how much luck is going to be in the outcome. So if you're thinking about, do you want to bet on that something is going to occur? They start to think, well, I don't know, let me think about how much... Like what's the range of outcomes, how common are these going to be? So it actually causes that uncertainty to bubble to the surface, that influence of luck to bubble to the surface.

But the other thing it does, is it gets you to start thinking about why do I know what I know? How well do I know it? Like how sure am I? Because that's really kind of what someone's challenging you to. And one of the things I think that want to bet does really, really, really well, if we think about how do we figure out what our own state of knowledge is, how confident we are in what we know? It's that we know that when we're thinking about our own knowledge and sort of what we see in the world, that we're very overconfident about the things that we know. And that's because we're looking at the world from the inside out. So when I say, want to bet, to you, I actually force you into the outside view. Because immediately, and you can correct me if I'm wrong, one of the questions you'll say is, "What does Annie know that I don't know?"

So it's essentially like a cue to, first of all question, why do you know the things that you know? So you start to sort of go through some vetting like, what's the evidence that I have for the belief? Where did I learn it? Have I researched it? But then it also starts to pull you into the outside view, what is the person who wants to bet against me know that I don't know? You start asking why am I wrong instead of why am I right? That's kind of really important. And you think what are other things that might be true? Or in the case of outcomes, if we were talking about the Pete Carroll thing and you said, "What an idiotic decision." And I said, "Do you want to bet?" One of the things that you would do is you would start thinking, well, what are the other ways it could turn out? Was that the only way it could have turned out? You might ask yourself, how often does an interception happen there? So you start to say to yourself, well, what's the range of outcomes and kind of what's the probability?

So what we want to do is recognize without always saying to ourselves, do you want to bet? Do you want to bet? Do you want to bet? How do we create good habits of mind around making sure that our beliefs are kind of as good as possible? So there's two things that we want to do, we want to audit. If we think about the box of stuff we know, we want to be doing some auditing of that. So that's the, why do I know this? How sure am I of it? Those kinds of questions. So that's kind of doing an internal audit. But then we also want to figure out what's the best way to extract stuff out of that box of things I don't know into things I do know? And along with that to say, how do I actually extract the maximum stuff that I don't know that I don't know? Because that's actually really important to do. So how do you do that?

Well, the first thing is you get focused on wanting to go find the information out, so that's number one. And the actual interesting thing that sort of spurs you to do that, and the hint is from, want to bet? If I say, "Want to bet?" It makes you start thinking about what's the range of outcomes and how often do they occur. Now notice, for you to do that you have to go think about your knowledge and then you might want to find some things out that you realize that you don't know. So, just the act of sort of thinking about, there's always a lot of ways that things can turn out and each of those ways has some probability of occurring.

And if I start to, first of all, take a stab at figuring out the range and then second of all, take a stab at figuring out what the likelihood is, which may not be an exact number, it might be a range, sometimes it would be like, I know the divorce rate is 50%, sometimes it might be, I don't know the exact divorce rate but it's somewhere between 40 and 60%, just the act of doing that starts to make you do these internal audits, and it makes you try to go find information that would inform that process. So that's number one, that's like the kick in the butt to get you to go do that.

And then the second thing that you have to do is figure out, I only know what I know, and I know that I am going to be processing the world in a way that is biased. No matter how hard I try, I might get a little bit better at it, but I am always going to be reasoning in a way that's going to support the beliefs that I have, I'm going to be reasoning in a way that makes outcomes make sense. Once I know an outcome, it's very hard for me to get out from under the shadow of it. So you can tell me if this is true for you, but I know the math of the Pete Carroll, I know it very well. I know that in, I think it was game four of this year's World Series, It was totally fine to pull the pitcher in the seventh inning. I know it mathematically and yet, if I really had to be honest with myself, like what's my emotional reaction to it, Pete Carroll's an idiot. So given that I know that, I've looked at it, I've done the math, I wrote it in my book-

Patrick: [00:31:54] You literally wrote the book.

Annie: [00:31:56] ... and still in my head somewhere there's a voice going, "Pete Carroll's an idiot, Pete Caroll's an idiot." So if I can do that in a situation where I've actually done the work, imagine how much this is happening without it being in view. And you don't even know how much your reasoning toward the outcome to make the outcome makes sense, reasoning toward the things that you already believe. So that's motivated reasoning where we think that we're processing information in an objective way in order to construct the most accurate beliefs. But what's actually happening is we have all these beliefs and they are determining how we process information in kind of two ways. The first is that basically we pay attention to stuff that agrees with us. And secondly, when we do have information that disagrees with us, we're very good at spinning a story either to discredit it or to actually slice and dice it, to actually agree with us. So we process information to support our beliefs, not process information to calibrate our beliefs.

Patrick: [00:32:49] This is the kind of like hearing is believing the idea from the book.

Annie: [00:32:51] Right. Exactly, exactly. So this motivated reasoning is very circular, we know that we're going to fall into this. And we can think about that even just in terms of resulting, which is as human beings, we're not very comfortable with uncertainty, we're not very comfortable with randomness. We want outcomes to make sense. We don't want to think that bad things can happen to our good decisions, that makes us sad. We don't want to think that if a good thing happened, it was a bad decision, if that makes sense, that would make us super sad. And so we're always doing this circular reasoning. So if I know that, then I know that on my own, I'm going to be not so great at this. So one of the things as I'm saying, I want to be better at these post outcome reviews, I want to be better at figuring out why things happened so that I could win more bets. That's the game here, right?

The game is like accuracy. The person who has the most accurate beliefs about the world is going to win in a betting situation. So this is now the game. So after outcomes, how do I create the most accurate view of the world? How, when I'm in the process of trying to decide, do I make the best decision and actually have the best view of what the possible outcomes are? How often they'll occur? I know I can't do that on my own.

Oh, okay, I can use other people because when I think about the box of things I don't know, other people know stuff that I don't know. When I think about what are other ways that things might turn out, you might have a different idea than I do. When I think about how often might those things turn out, well, you might have a different view than I do, you just have a different perspective on the world than I do. And if I were talking to you, there would be things that you might tell me that I knew that I needed you to tell me, but there's probably a lot of stuff living in your head that would be useful to me but I don't know it's useful to me, and I'd like to get those out of your head as well.

So how do we do that? So there's kind of three main tacts. Tact number one is to talk to other people including uncertainty in the way that you talk. So if I say something like, "Obviously Trump's going to lose the election in 2020." If you have other ideas about that, I haven't left any room for us to really have a conversation about it. Either we're going to get in a battle about it because I've declared that this thing is true 100%. So either I'm going to just get out of you, "Well, you're wrong and you're an idiot," and we're not actually going to have a real exchange or in a cocktail party situation, you might not even speak up. And you might have a different view than I do, and I now don't have the opportunity to learn that because I have declared something with certainty. I haven't left the door open for us to have a rational exchange of information about why you might view the world differently than I do.

But if I were to express it instead as, "Yeah. I'm kind of interested because I've seen a lot of stuff and I see different opinions about what the likelihood that Trump will win in 2020 is and I'm just not really sure. I think he's probably an underdog, but maybe he's like 47%." Now that we're not having a disagreement, where's the disagreement now? So I've now let the door open for you to now start offering me up a whole bunch of things that you think about it. You could say, for example, "Well, I actually think the percentage is higher." And you haven't told me I'm wrong because I didn't declare right or wrong in the first place. You're you can say, "I think this is the usual mistakes, I think it depends on which candidate..." Now, all of a sudden, you're going to have a conversation with me about it.

And you can do this in anything. Don't declare, "I know this is the right strategy for investing in the stock market," Be like, "Well, I'm sort of looking at it." And there's all sorts of different ways that you can express uncertainty. So I can say, "I think this is a really good route to take to the airport, I'm like 76% on that." I do this when I'm writing a book, I'll say to my editor, "I think that I'm going to have the next chapter to you on Monday 54% of the time, by Friday it'll be a 92% of the time." And she actually has told me back that she actually really enjoys that because it's really good expectation setting, it gives her an opportunity to probe as to why Monday is 54%? Could she support me in some way that could like speed it up? Why is Friday only 92%? What's going to happen after Friday? But it gives people room to do that. So that's kind of strategy number one.

Another thing that you can do is, "I read this opinion piece, I'm really interested in what you have to say about it. But I just want to let you know, it's the only thing I've read on the topic." So now what I've done is I said I have an opinion about it, here's what I think about it but I just want you to know it's not that well informed of a decision which now lets you know that I'm pretty uncertain about what my opinion is and I'm just looking for information. Now I'm just speaking in a way that's more likely to be a good information extractor out of your head.

The other thing I think that's really important to understand is that in order to be really good at this knowledge piece, like how do I think about why the world is the way it is? How do I process my experience? How do I learn from it? How do I think about what a good decision is? What the outcomes might be? I have to be really good at how do I get feedback from people? And the number one thing to understand about getting feedback from people is that they have the same biases you do, and they will reason toward beliefs the same way that you do, and they will reason to make outcomes make sense the same way that you do.

So it's really important to understand that your beliefs are infectious in a bad way, they're contagious, they infect the other person, and that the way that the world has actually turned out will also infect the process. And that it's not just you talking to other people, but it's anybody who gets in a room together. They're going to start to infect each other with their beliefs, they are going to start to infect each other with the outcome.

So, for example, if I'm deconstructing a past decision, it would be really good if I'm trying to get your opinion on it, if I sort of work through the decision only up to the point that I needed to decide. For example, if I'm trying to say, "Hey, do you think this is a good play in football?" I would say, Well, it was on the one yard line and it was second down, and I only had one time out. And I could hand it off to Marshawn Lynch or I could throw it. I was down by four, so I needed to score a touchdown. Obviously I don't have a lot of time and I only have one time out, so here's what I considered." Well, first of all, I could stop there and not even tell you anything that I considered or I believe. So that would be pretty good, what do you think?

So now I'm not infecting you with my beliefs at all and now I get what you think, but maybe you need to know some of the math, maybe you don't know that. So what I would tell you then is, "So I really felt like I wanted to get the ball to Marshawn Lynch twice. And as I was thinking about that, I kind of thought, well, if I hand it off to Marshawn Lynch the first time and he doesn't score, a ton of time is run off the clock so I have to call a time out, then I hand it off to Marshawn Lynch again, that's the last play. So then I thought, well, maybe I should pass because if I pass and it's incomplete, the clock stops, I can get it up to Marshawn Lynch again. What do you think?"

So notice, I haven't told you that it ended in this disaster. So when I describe it to you that way, you're now by definition out from under the shadow of the outcome and you can actually see through to sort of what my thinking process is and what my decision process is a lot better such that I'm going to get higher fidelity feedback from you. So you can do this in two ways, you can make sure that you're doing a lot of decision review before you get an outcome. So I say if you're a trial lawyer deconstruct your strategy before the verdict from the jury because that one verdict probably there's not going to be a ton of signal in it, you would have to sort of see across a lot of different... There's a lot of randomness there. But if the verdict has already come in and I'm going to talk to somebody else to try to find out what they think about my strategy, I should never tell them what the verdict was. I should act as if it didn't happen yet so that I'm not soiling the way that they're going to now think about the decision.

So that's number one, it's like quarantine outcomes. Number two is to actually quarantine beliefs from each other. So that would be that second piece of the Pete Carroll thing. I can say, "Here were my different options, I could pass first and then if that failed, I could run the ball twice. I could hand the ball off first..." So I can just walk through, here where the different options under consideration, and then not tell you which one I like. Because once I tell you which one I liked, either I'm putting you in a position where you have to disagree, or I'm probably going to be pretty convincing, and you're not even going to know it, but you're going to start looking through the frame of trying to justify the thing that I've told you that I believe.

And it's going to depend a little bit about whether you sort of consider yourself in tribe or out of tribe to me. If you consider yourself in tribe, you'll reason toward my belief, if you consider yourself out of tribe, you'll reason a way from the belief. We can see this in politics all the time where it's that thing of, what if Obama did it or what if Trump did it? Well, that's silly because that's a tribal problem, not a belief problem. So I can just literally present you with the different options that I had, and then not tell you which one I chose, not tell you which one I believed was better. And now you can think about how I could do that prospectively. When I'm considering a decision, I can tell you the options that I have and now I can say, "So what do you think?" And one of the things that I can ask you about is do you think that I'm missing an option? That's a good question to ask. And now I haven't told you which option I like, so now I'm going to get much better feedback from you.

Then the third piece of that puzzle is to make sure that you're doing that when you have any kind of group decision making going on. So if I'm trying to elicit feedback, I want to actually elicit it separately. So I think the easiest way to conceptualize this is if you have a hiring committee, to have each person interview the candidates separately and then have them each write their debrief separately before they get in a room with each other.

Patrick: [00:42:21] Because then group think sets in.

Annie: [00:42:22] Group think, but sometimes anti-growth think. I think that one of the things, again, we need to remember is that sometimes if someone views somebody else as out of tribe, that they'll actually go in the other direction, so it's not just a group think problem. Sometimes it's like, "You're not my team, so I can't agree with you." And that's just as bad, that's not good either.

Ego Suppression

Patrick: [00:42:40] So much of what you've said makes me think about, and back to your psychology roots, studying psychology as a student, about this concept of ego and that in many of the bad influences on decision making, a large ego seems to be getting in the way. So do you think that one of the best strategies for improving decision making is to suppress the ego, to kill the ego?

Annie: [00:42:59] I might give you a surprising answer here. No, let's go back to the beginning of the conversation, what are the rules of the game? I want to win. Here's the thing, I think that we're not particularly well built for absolute pricing. When you think of price of goods, price of goods, it's relative, there's some at the base, there's like cost to produce. How much should this pair of shoes versus this pair of shoes cost? It's a lot like, what will the market bear? What are people telling you they're willing to pay for it? What does this look like compared to this other thing? If they're very similar, the price has to be similar, relative to each other, despite what the cost to produce might be. There's all this sort of relativity just in cost of goods but you can think of also just in terms of your own happiness. We know that if you ask somebody, "Would you rather be in the top 10% of the socioeconomic status in 1650 or in bottom 20% of socioeconomic status today?" Literally everybody chooses 1650.

For absolute pricing? Like, why are you choosing 1650? I think your life expectancy is like 40 or something, there's open sewer, there's no vaccines, what? But because we don't really know we're happy unless we're comparing ourselves to other people. So I think that suppressing your ego is an amazing goal, I really do. I practice mindfulness, I try to do that. I think that we're doomed to fail because we don't really know how to value ourselves without thinking about, how do we value ourselves compared to other people? And I want to be clear that I think you should practice that other thing, I think you should practice, how do I take my ego out of it? How do I not have that? But I don't know if you've ever talked to certain people who practice this stuff, but they have a lot of ego about how much ego they don't have. Have you noticed that?

Patrick: [00:44:49] Yeah.

Annie: [00:44:49] "Oh, I'm very mindful and I really take my ego out of everything. I'm the non-ego guy. I'm just way better at not having an ego than you." People should listen for that because they'll hear that there's all those people who are like, "I'm better at not having to go than you. So I'm sort of like, "Okay, I'm going to practice trying to take my ego out of things. I really am." But I also recognize that I'm going to end up being one of those people who's like, "I am way better at taking my ego out than you."

So what do we do then? It goes back to the rules of the game. You identify, these are the things that I value, here are the things that I think are going to make me better as a decision maker. One of these things, when we think about the anatomy of a decision, we know that at the base of it is the stuff you know and the stuff you don't know, doing that constant internal auditing of that stuff you know box and then trying to extract as much of the stuff you don't know out of that box. So I know that I have to be doing that. And in order to do that, I have to be willing to let go of my belief sometime, I have to realize the thing that I believed was wrong. I just actually had this happen to me, it was a really wonderful thing.

Sometimes I have to calibrate my belief a little bit. Now I could calibrate it in two different ways, that my belief could become stronger, I could think the thing is more true, sometimes that's the right direction to go. It could kind of stay the same, I could believe it a little bit less, like I could go from 72% on that and now I'm 63%. But sometimes I can have a full on reversal and those are the things that I should actually be the most thankful for. Because if I really had it wrong, every decision is informed by that belief that I have that's like completely out of whack. So obviously my decisions are only going to be as good as my beliefs. So now if I say, for me, the winning game is to make better decisions, and at the base of that is that this box has to be more accurate. And this box of stuff I know, these things that are in it, in order to create more accuracy, I have to really be seeking out this stuff I don't know box because that's what's going to inform those. Now when I find out I'm wrong, I'm winning game.

And by the way, I feel pretty good about myself that I'm winning the game, and I feel pretty good about myself that I changed my mind, and I feel pretty good about myself that I gave that other person credit, and I'm really thankful that they disagreed with me, and I'm really good at that game. So there, I don't think that's so bad.

Improving Decision Making Ability

Patrick: [00:47:05] Yeah, that makes a lot of sense. When I say, think of somebody that stands out as being exceptional at improving their decision-making ability that you've had in your life, who comes to mind?

Annie: [00:47:15] I would say that the person that I know is the best at this that I've ever seen as Erik Seidel, a famous poker player. He's just really good at seeing where he might be making mistakes. He is so focused on that, very, very open-minded, really wants to learn from other people. I think it's very hard when you're sitting at a poker table and somebody beats you, it's really hard to just admit that they might've played better than you. It's particularly hard when they're doing something that seems weird to you, that's a strategy that you hadn't thought of before, that you might not be in your wheelhouse, then you sort of feel like you had a deficit.

So it's a lot easier to sort of swat that off and say they're just a lucky idiot. And he's just so good at saying, "I'm the idiot." He's so good at that, and I think it shows in his results, shows in his temperament. He's very calm, he doesn't get upset too easily, I think he really defined what is the goal for him so early on, and then I feel like I was just really lucky to come in contact with him. I got to catch a little bit of that disease. I'm not anywhere close to him, he would be like my goal, getting to where he is would be my goal.

Patrick: [00:48:21] People of that class that are good at improving their decision making ability, can you say anything about their tendency or lack of tendency to change the rules of the game? So how often are people that are really good at this, playing new games versus just getting really good at some original thing that they're trying to accomplish and being consistent?

Annie: [00:48:38] That's a very interesting question. So I think that at a top level, you have to recognize that once you identify what a goal is, which obviously is informed by your values, that you can figure out what the rules of that game are for getting there. In that sense, you always have a singular focus on winning the game. I always have to figure out what my goals are, given what my values are, what does that goal mean for me? So that's back to that question of, if I know my goal is to provide for my family, I have to think about what my values are so I understand what provide for my family means for me personally. And then I can sort of go backwards from there, I obviously think backwards time traveling, I talk about it in my book, is really important. What are the structures that I need to set in place? Like what are the contracts that I'm making which really defines what the rules are and what it would mean to win that game.

So I think for people who are doing this, they recognize that very fundamental, basic thing. And then what I think that they do is, situationally speaking, they then now use that same strategy in order to figure out, given this particular thing that I'm engaging in, what are the rules of the game? When I'm really frustrated with someone who's like a customer service person, generally, I don't yell and I don't get upset because the rules of the game are actually to get out of them what I want. And if I yell at them, I know that I'm just reducing my odds of doing that. So I tend to be very calm in those situations. Not because I'm actually calm internally, I'm upset, but because I've defined very quickly what my goal is and what do I need to do in order to win this game. And yelling is very counterproductive to that, even though I'm really mad.

Patrick: [00:50:03] Really clarifying answer. It makes me you want to think of one of my favorite books, James Carse's Finite and Infinite Games. Have you read that book?

Annie: [00:50:09] I have. Yeah.

Patrick: [00:50:10] So I think an interesting way to frame your answer is that the value based game is sort of the infinite game that he advocates you play. I think if I hear of Dreams of Sushi or something, I love that framing of really the game starts a layer before the rules of the game. Kind of like we've talked about where you're switching and it's situation dependent and everything else. The last topic I want to explore with you is the role of psychology when you're dealing with other people. So when other people are a part of a decision that introduce sort of a big error term because of human psychology. And this idea that you have in the book that life isn't chess, it's poker, because people are involved and there's always sort of the math component and the people component. So using poker or not, maybe just riff on that concept of how to deal with the psychology of others and factor that into the sorts of decision that you're making which very often involve others.

Annie: [00:50:55] Let me just start with a little chess versus poker because we've kind of talked around it a little bit. When we were talking about the anatomy of the decision, we said there's kind of two big things in there. The first is stuff you know, stuff you don't know, and we can think about that as incomplete information or poorly formed beliefs. Because if we think that something is true, that's not, that would then go into the category of incomplete information. So we have incomplete information and then we also have this intervention of luck in between the moment that we make the decision and whatever the outcome is, so there's an intervention of luck. And we think about luck generally as stuff we don't have control over. So if we think about chess, there's a little luck, but there's not really a lot, certainly not in the sense that somebody rolls dice and then your queen comes off the board or something, oh no, it's a seven, bye bishop. And there's not hidden information in the sense of I can see the whole position on the board.

Why that's important it means that pieces will only move by an active skill. And because I can see the whole position, I know all the possible places that those pieces could move are. So what that means, just in a simple thing, is that resulting in chess is a totally reasonable strategy. We play a game, I lose, we know my decisions were worse than yours. By the way, back to relative pricing, we don't know in an absolute sense how good my decisions were, I only know compared to you.

So now if we think about poker, that's no longer true. There's hidden information, I can't see the cards, and there's also luck; cards turn, stuff happens, you beat me. So in poker, now we can see that resulting is quite a poor strategy. We play some poker, an hour of poker, I lose to you and we don't really know anything about my decisions in comparison to you, we just feel like we do. We feel like we know I was the worst player, but we actually don't know that because in poker I can have the best hand, I can play it quite well and I can still lose and vice versa. In chess, that just doesn't happen, if I make the best moves, I get to win, so that chess versus poker thing and poker is just a better model for decision making because of that.

Now, one thing that happens in poker is that the hidden information is really what do you have? But then we can also think generally about not only that your cards are hidden from view, there's also things that you know. So you know what your cards are, but then there's also, how are you going to react to your cards? So the actions of others are also things that I can't control. There are things that I don't know, I don't know what the actions of others going to be, I don't know what your thoughts and feelings are really, I don't have access to that, I hardly have access to my own to be fair as we know from psychology, but I certainly don't have access to yours. So it's not just that I don't know what card is going to come out, it's I don't know how you're going to act and you're not necessarily always going to act rationally. So if I knew you have aces and Sally has aces over here and I bet you might do something with those aces than Sally will. So, I don't really know that and I'm guessing at that.

So obviously when we're talking about poker, I'm now trying to in a zero sum way, best you. So I'm trying to take a guess at what are your cards and what are you thinking about them and how do I think about what you're thinking about them? And this is all meant to be an even exchange when you lose hopefully or sadly, maybe you win, I lose, how good I am at kind of narrowing down those ranges, like working in those uncertainties, identifying how much luck is going to be in the outcome, taking a better guess than most people at what you might do with your cards, how I might be able to get you to sort of act in certain ways, what the probabilities are of the different hands that you might have, that's going to kind of determine how I win. So that's the straightforward stuff that we've been talking about of how good are you at figuring out what's in that stuff you don't know box? How good are you at auditing the stuff you do know box? How good are you at figuring out what your options are?

So obviously in poker, it's limited, I could raise, call, fold, how good are you at figuring out what the possible outcomes are given any of those options that I take? How good are you at thinking about how often those things occur? So that's relatively straightforward. What starts to get more complicated is that very often we aren't the final decision maker. So sometimes we have a boss, sometimes we have a committee that we have to get to agree with us, we have children. So the question is, how do we balance out? I want to be kind of making the best decision for myself, but then I've got other people who have influence on what my outcomes are. And I don't really have a lot of control over them, so how can I then behave in a way that kind of makes that all work a little bit better together?

And we touched a little bit on it, number one is any kind of committee situation, keep people away from each other. The other thing is the way that you talk to people can get somebody who otherwise would not necessarily be somebody who would then become a thought partner with you to be a thought partner with you. So the less that you're using words like wrong or right, or whatever, you're going to get much better information out of people. And one of the things that I really advocate is to never ask someone, are you sure? Ever. To only ever ask, how sure are you? And if you just make that little switch in language, you will get them to be a partner with you, even if they're not necessarily so inclined. So I think that that can be really helpful.

I think the other thing that can be really helpful is transparency. So if we think about part of the problems with resulting and hindsight bias and all of these things where we have these kind of out sized influence of outcomes, if we think about it, it's really being caused by what's the transparency of the decision itself. I can show you what the transparency is pretty easily. When there's a decision that is transparently good or bad, we don't resolve. So I'll give you an example, "Hey, I was driving along at the speed limit with a very well-maintained car and I proceeded through a green light in very good weather and someone came through the intersection and T-boned me, did I make bad decisions? No, but the outcome was an interception, It was a terrible outcome.

Now, why is that so easy for you? Because the transparency into the decision, you know what the quality of that decision is. So now we can use that to our advantage because people say to me, "Well, this is fine. I can do this stuff, but what do I do about my boss? My boss is really outcome oriented and I know if I have a bad decision, it's all over." And I said, "Well, I'll be more transparent about the decision with them." When you have a decision, actually, you try to build out these kinds of trees that we've been talking about. Here's the decision, here's the stuff that I was thinking about, this is kind of what I'm considering, here are what I think the options are, tell me what you think.

And if they say, "Well, it's your job to tell you what I think." Okay, well, here's what I think and here's why, but I want to give you the opportunity to tell me why I'm wrong. Tell me what I'm missing, is there a piece of information, is there like an outcome I'm not considering, am I not taking into account risk of ruin well enough." Whatever it might be, just show them that. And the thing is that, interestingly enough, they may not even engage so much. But the fact that you showed it to them, later, if you have a bad outcome, you can be like, here, we talked about this. So I think that transparency is really, really good.

The other thing is to accept that not all decisions require a consensus, you can work through a process and you can still have people who disagree with the decision. But if you've included their voices into the decision, they'll usually feel pretty good about that. And you can do that in a few ways, you can obviously red team, if you do pre-mortems which is imagining failure that allows the dissenters to actually have a voice in the decision process. And I don't know anybody who's ever done a pre-mortem who doesn't then do some changes to the strategy so they feel incorporated. Once you reach consensus, you can do things like say, "Well, this is alarming, let's go figure out why we all agree with each other. And then all those people who maybe wouldn't otherwise be speaking up, get to say so. And even if you do end up going with that decision, they feel like they've been heard, and so that's also helpful in group decision-making.

And then at some point you also have to say, "I can't control other people, I can just do the best thing for me and that's okay too." Just sort of like understand, there's things I can control and things I can't. And being exasperated because someone else doesn't want to see what I see, well, first of all, I should ask myself, maybe I'm exasperated because I'm not being open-minded to the fact that what they see actually has some value, you should always ask yourself that. But even if the answer is no, I've really thought about it and I've done this in a way that I think is pretty rational and I think that I should just be exasperated with them, why are you exasperated with them? You don't have control over them.

Patrick: [00:59:19] I think the ideas of working with people better and maintaining uncertainty it's one of the major things I'll take away from this, never closed the doors. Just leave communication pathways open when we're so prone to close them, right? We so want black and white outcomes like determinism, and that's a huge mistake for making good decisions.

Annie: [00:59:35] Exactly. And if you think about the person that you're really exasperated with, you can be exasperated or you can say to yourself, "Okay, I'm exasperated with them, is it so much so that I'd like to fire them as a client?" So maybe that's the answer, maybe they are such a time suck. Because they're calling me always, like they can't get it into their head. Stop looking at what your balance sheet is, please don't look at your P&L, stop. Maybe that's the answer, but maybe the answer is, "Oh, this is interesting because they're super exasperating. But let me see, maybe I can move them. How can I change the way I talk to them?"

Take it on as a challenge for how you can learn, I think this is a good lesson from poker. There's people acting irrationally at poker all the time around me and sometimes I do something where any thinking human being would fold and that person calls me and my bluff doesn't work. Well, frankly, it's up to me to figure out how I don't make that mistake again, because that's my mistake that I didn't recognize that they weren't going to act in a way that I thought they were "supposed to". That's on me, as opposed to just dismissing like, "Well, they're just dumb."

So if you have someone who's exasperating, figure out, can I move them? Can I speak to them in a different way? How can I change my strategy to get them into a place that I think is going to be both better for them and better for me? And sometimes the answer is, it's not worth the time, but mostly you're going to learn something from that. And then what you'll find is that having taken the time to approach with a more open mind the person who you view as so irrational and so exasperating and trying to really say, "Well, here's the rules of this game, how do I get into the best place with this person?" Is that then you're going to learn some things about what worked with that person that you can then take into somebody else that you might be working with.

And you're going to figure out they really responded when I phrased it like this, they really responded when I used this particular identity language, they really responded when I compared them to this group of people, they really responded to whatever, and then you're going to see that with the most exasperating person, with the person who is most prone to being irrational, most prone to being reactive, most prone to being emotional, whatever it might be, that you found a way to damp that down, that then it's going to improve every single relationship you have with everybody else. And you only did that because you didn't say that person's just an idiot and blah, blah, blah, as if you had no part in that relationship.

Patrick: [01:01:56] So I've got a couple of closing questions for you that are a little more fun. So first question is, what's the best decision that you've made?

Annie: [01:02:01] I imagine I don't know what it is. That's just sort of the nature of it. And I hate saying... I can say the best decision I made is to have children, but how can I possibly say that because who knows what it's like to have children beforehand? So I can tell you that's a decision that worked out really well, was having children. It's hard for me to say that here's what I think the best decision I made because this really was a conscious decision.

I actually tell this story in my book, when I first started playing poker, my brother had taught me and he'd given me like a pretty simple list of hands that I was allowed to play. So I went out and played and when there were people who were not following the list, I thought they were dumb. I really did, I was a jerk. I wasn't saying it out loud to them like, "Oh, you're an idiot." But the me in my head was like a total freaking jerk. And I felt very superior to them because, of course, I knew how to play and they didn't, and I was incredibly dismissive of them. And I sort of think back on that and I'm like, I could've gone and asked the person who gave me the list, that seems a little weird. But I was just very sure that clearly my list was really good and these are the hands you had to play, and if these people were playing a hand different than what was on the list, they were just idiots.

And sometime into that, a year after I started playing, I realized, wait, maybe they're not idiots. Because I started realizing as I was playing, I started figuring out, well, sometimes actually there's certain situations you would play different hands, like the circumstances demand it, that's sort of what bluffing is about. So all of a sudden, I remember looking back and saying, "Oh my God, I've been like dismissing all these people's as idiots." And I suppose some of them were, maybe, but not all of them. And it's that thing, that Erik Seidel thing, I realized, "Oh my God, I'm the idiot here. I'm the idiot because I have just been dismissing them and I haven't been trying to figure out is what they're doing okay."

Why? Why are they doing that? And there's two things that come out of why they're doing it. Thing A is that's actually a reasonable thing to do in that situation and I can think about that and maybe incorporate into my game. Thing B is I understand why they're doing it so I can recognize when it will happen again and then I can counteract it. Both of those things are very, very good for me. So here's the decision I made, I think it was the best decision I made in my life. I said, "I am going to try to figure out the thing that every single person at the table that I come across does better than I do." I'm not saying that that was my best decision because it happened to help me in poker, it was my best decision because I think it really changed my life. And I would say that's the best decision I made.

Patrick: [01:04:24] My closing question for everybody is for the kindest thing that anyone's ever done for you.

Annie: [01:04:28] I've had so much kindness in my life. I think if I have to pick like a isolated moment that doesn't have to do with the usual stuff... I'm trying to give you something other than the usual stuff because my husband is the kindest person. He is so kind to me that I'd have to write a book about the kind of.... He wins like all the kindest things. But I feel like that sort of in the normal category, so I'm going to give you something that's outside the norm, a very kind thing that somebody did for me that really had an impact. When I went to graduate school, I had an amazing advisor, Lila Gleitman, and I also worked with her husband, Henry Gleitman, but Henry's passed. And Lila was an incredibly important person in my life. She's brilliant and funny and incredible, and she was really like a mother to me. She wasn't just like my thesis advisor, she was really amazing.

At the end of graduate school, as I was going out for my job talks, during the last year I've been struggling with this very bad stomach problem which we had discussed before and I was really sick. And I was going out for my first job talk at NYU and it just got to the point where I was so sick I just couldn't keep food down and I ended up in the hospital for two weeks, I had something called gastroparesis. So I was literally on an IV, it was pretty bad, I was really sick. So in academics, the job market is seasonal and so I had to cancel all of my job talks and then obviously I needed time to recover, it was a very bad time in my life. I felt like this was all very unlucky, which is a good lesson in itself because I think it was actually quite lucky that my life took that turn.

So now I take time off from school and it's during that time that I start playing poker. So the thing about graduate school is that it's like five years of your life that you're putting this time in and I felt very keenly that this was time that these people had put into me and like a tremendous amount of effort that these people have put into me. And as I was kind of going through this physical crisis and I start playing poker, I think that I sort of felt this need to like escape from the world a little bit. And at that time poker was not on television, it wasn't until about eight years into when I was playing poker that it became on TV. So it was a very good vehicle for hiding from the world a little bit because I physically didn't feel well, and so I think that's kind of what I wanted to do.

So part of that was that I sort of hid from them. I had some contact with them over the years, but not too much. And I have a probably overdeveloped feeling of shame in disappointing people, and so I really felt that, and then that caused a problem with me sort of continuing to interact with them. I carried through my whole life as I was having this flourishing poker career, and then about eight years into I was playing poker I actually started doing the academic work again through merging the two. In 2002 is when I started doing the work that you see in Thinking In Bets. I just always felt like I had really disappointed these people, and in particular, that I very much disappointed Lila, like really disappointed her. And I was very ashamed of it, I imagined that she was quite ashamed of me, and this was all kind of like this narrative that was living in my head.

So we lose touch, really my fault, completely my fault because I think that I just carried around this sense of having failed so strongly that my fault I hide from her. So now this amazing thing happened. In 2012, I tore my bicep tendon. At this point, I've retired from poker, I've moved back to Philadelphia. And I'm in an orthopedist office and I look over and there's Lila. This is the kindest thing that's ever happened to me because I walk over and I sit down and I say, "Hi, Lila, it's Annie." And I think I use my maiden name, I think I said, "It's Annie Lederer." And it was just this awe and smile and "Oh my God," and hug and sitting and talking. And then she just said, "I'm so proud of you." And we exchanged information and we have had lunch basically every three weeks ever since. And it's like I have this mom back.

But it was that moment. She never said to me, why did you leave? Why did you stop contacting me? Why did you lose touch? She could have foisted so much on me about my not having maintained that relationship, and there was none of it. It was just, "I'm so proud of you." And that is absolutely the kindest thing that anybody has ever done for me.

Patrick: [01:09:01] One of my all-time favorite answer answers, such a great story. I always love the answers with a theme of, forgiveness, may not be the right word, deep understanding. It's pretty powerful.

Annie: [01:09:09] Yeah. I went over with trepidation because I was thinking she's going to be like, "You asshole, where'd you go? Like, where did you go? Why did you run away?" And hopefully people listening who've struggled with something physical will understand that desire to run away, and then once you've run away, it's how do you recontact because you sort of feel the shame of having withdrawn from the world and withdrawn into yourself that really messes your thinking up.

Patrick: [01:09:37] Well, wonderful answer and just such an interesting, amazing conversation. So thanks for all your time.

Annie: [01:09:41] Well, thank you.