The study has be telephoned to death by the popular media thanks in no small part to the snobby gatekeeping mentality of the wine industry, whom people just want to say are full of bull. (I say this as some one with an amateur interest in beer, wine, and spirits. Someone with a decent palate who regularly does blind tastings)
This analysis is exactly correct, IMHO (or my trapped priors lead me to think so, anyway....) I think the fundamental question, unaddressed here, is "How do you catalyze people into recognizing, accepting, and acting on the fact that they have a trapped prior?" Anyone? Bueller?
The mechanism doesn't seem like it has to lead to irrational behavior 100% of the time. If it happens to correspond accurately to an expected value assessment (dog bites are unlikely, but really really bad; believing dogs are bad prevents dog bites; dog phobia is justified from an expected value standpoint) it might be functionally valid even if it's epistemologically invalid. Rational behavior is more important than rational belief, and there's no reason they have to correspond.
The analysis here seems correct and incredibly important. If we could get people to recognize this concept of trapped priors within their own experience, maybe it could help us break out of this emotionally charged era and lead to greater cooperation and less conflict.
The problem as it appears to me is, how do we communicate this idea to the public? The way you've presented it here works well for communicating with people who tend to view concepts in rigidly logical ways, but I think this approach will fail with the general public. I'm thinking of my wife, who is a therapist who has done a lot of work with trauma. She has said essentially the same things presented here in our conversations, but she frames it in a way that is rooted almost entirely in personal experience. If I were to try to bring up the concept of trapped priors this way I don't know that it would click with her, or even if she did agree with the theory she might reject this way of thinking about it as unhelpful to her clients with trauma.
So how might we present this in a way that is easily digestible?
Maybe other people have already thought about this and found ways to present it to a general audience. I'd be interested to know if anyone has resources like that. I think the inability of science to communicate its results effectively is one of the greatest barriers to progress.
Brings to mind a conversation I had with a politically radical relative, whose extremism on politics was noticeably ratcheting up at the time. I shared with him a study about how easy it is to get partisans to agree with false statements by presenting them in an appropriately partisan context. He responded by explaining at length how the explicitly false statements on his side of the issue were actually *true* and the study just proved even more how right his political beliefs were.
Very educational conversation for me... Changed my view of political belief a lot.
I definitely notice my strong prior against assertions made by certain parts of the political spectrum, but I'm not sure how much to fight that - if my prior is "this person or group has lied or misled in a large portion of the things I've heard them say" that seems like a useful, rational thing to allow into my judgement of new assertions. Or maybe I have a stuck prior which leads me to misinterpret the things they say, thus making them seem to be liars? Bit of a rabbit hole.
The common thing that is often mentioned in the context of talking people out of an epistemic "echo chamber" or out of a radicalist cult, or even just out of garden-variety homophobia, is getting to know a friendly person who is from the outside, and gradually coming to like them and eventually trust them and listen to them. This really sounds a lot like the desensitization therapy that works - it's never going to help if Ted Cruz or AOC themself come to get to know someone and show that they don't bite, but if someone who seems nice and isn't already processed as a political symbol gradually reveals their niceness while also gradually revealing their partisanship (or the fact that they're gay) then it can overcome the defense mechanism.
A cute example for me was this sentence, from your post:
"Your prior that they're bad has become trapped."
I couldn't look at this sentence without seeing a typo. The two words "your" and "they're" in close proximity told my brain that one of them must be wrong. Weird.
as far as the relationship with political polarization goes i think this is a good argument for policies which increase diversity. if you have a trapped prior about people of type x being exposed to lots of people of type x will be required to overwhelm your prior. american history x is a great dramatized version of this. or daryl davis' stories. so ubi/remote work so that urbanization decreases somewhat, or maybe even something like mandatory federal service.
There is another factor that comes into play when this relates to politics. For someone with cynophobia, the material they are being presented with is actual evidence: whether it is pictures or direct experience with a dog. But in politics, the vast majority of the discussions do not revolve around evidence, but on the reporting of evidence. Nobody outside masochistic bloggers and other scientists are reading the original scientific studies about epidemiology, sociology, climatology, etc. They are reading reports and summaries about the studies produced by think tanks, academic institutions, professional societies, politicians, or more often the media. So it is not evidence that a voter is being presented with, but reports from intermediaries. As long as everyone trusts those intermediaries to the same degree, then the argument holds, but if someone has priors about the agenda or trustworthiness of the intermediaries, then that becomes the relevant prior.
And it's not just the prior about trust in the reporting, it is the prior about what you do next. Let's say you are a staunch Republican who believe that climate change is an overblown issue and being used by the left to advance their political agenda. You hear about a report that climate change is a problem. Even if you think of yourself as a rational person who wants to keep an open mind, you look into that report, and find out that it was produced by a group of scientists at an Ivy League university in collaboration with scientists in Europe, and has already been trumpeted as a clear argument for banning fossil fuels by a liberal organization. So at this point, before you've even had a real chance to read a sober account of the report by, say a neutral expert that you trust (assuming that one exists), you've already had 5 or 6 different priors triggered, all reinforcing your skepticism.
It is not just the encounter with the evidence where you engage your prior, it is the other 10 encounters on the way to the evidence that already have associated priors. It's like a person with cynophobia being told they are going to be taken to a room with a cute puppy in a cage, but on the way to the room, you can hear dogs barking, smell the dogs, and see leashes and bones scattered on the floor. It is a really hard thing to then still walk into the room. To provide a clear path to consider evidence that had political implications, you need a trusted source to explain it; one that you can turn to before the dozen partisan - or perceived partisan - entities can yell you back into your priors.
As far as I know the current view on exposure therapy isn't only about gradual exposure. It is in addition about getting the patient to make specific predictions what he/she expects to happen during the exposure and to try observing this as closely as possible. This could be worth a try with political biases, too.
Have another stuck prior, from a book called _How to See Color and Paint It_, by an art teacher who would have his students cut windows in 3" x 5" cards so it would be possible to just look at small areas. Then he'd have them go to the bay at dusk and ask them what color are those (brick) building's across the water?
They'd say "red", and then he'd have them look through the 3" x 5" windows, and the buildings were actually blue in that light. The prior for how the buildings look in typical lighting overwhelmed the actual sensory experience.
My experience with cynophobia fell somewhere in the middle of the treatments. For most of my life, I've been really scared of all dogs. I can keep it under control by expending mental effort, but it's taxing. Then, a few years ago, the ceiling of my apartment collapsed and the landlord refused to fix it. My best recourse was to move in with a friend in the area. The only problem? Another person living with him had a dog, the most golden retriever golden retriever I can recall meeting. I'd encountered this dog before, and could tolerate it, but only with effort.
Still, with a choice between staying in a room with a collapsed roof or a house that had a dog, I chose the latter. Eventually, I stopped freaking out when the dog started barking as I came to the door, and came to actually kind of like it. Months into this arrangement, I visited a friend of another friend who had dogs that were not golden retrievers. Two pit bulls, which he cheerily informed me had recently killed a groundhog in the back yard. And I was able to handle that visit without getting particularly stressed!
I think my cynophobia wasn't quite severe enough to trap my prior beyond where innocuous experiences could shift it, and I had a strong exogenous factor — wanting to get out of an unlivable apartment and into a really cheap and nice room — that motivated me to adjust my views.
In the past few years I have intentionally cultivated more trapped priors. I was directly inspired to this by an old scott post: Epistemic Learned Helplessness or something to that effect. On certain subjects I know that a) I am not equipped to tell truth from falsehood in this domain; b) lots of people around me are actively trying to deceive me; and c) they benefit from my deception. Consequently, my defensive reaction to this is to categorically write off everything as fake in that domain, and fall back on my priors.
I have cultivated this intentionally as a response to a common failure mode of rationalists, where they are willing to extend charity far beyond where it's warranted, and bad-faith actors use this to manipulate. I am not getting manipulated anymore.
> I don’t know why this doesn’t happen in real life, beyond a general sense that whatever weighting function we use isn’t perfectly Bayesian and doesn’t fit in the class I would call “reasonable”. I realize this is a weakness of this model and something that needs further study.
Since I hang out with engineers all day, I'm primed to think everything is feedback loops: Could we conceptualize the "trapped prior" as a problem in which an output signal (e.g. dogs are scary) is bleeding back into the input channel? The weighting function might be fine, but the problem is what counts as input? Would this help us identify a biological basis for the phenomenon?
I'd been citing "They Saw a Game" for this, which I think makes elegant points about bias shaping perception. But at the end suddenly seems to conclude that we live in some kind of shared multiverse with mutually inconsistent realities just stacked on top of each other as the most likely coherent explanation for "sometimes people disagree about stuff." So it's nice to find one less committed to the "what even IS reality man?" trappings.
For "the rationalist project"... who is the primary target for this intervention on stuck priors?
Do you want the average ACX reader to be more aware of their own trapped priors and find tools to overcome them? Or do you want to find a better way to talk to passionate partisans, maybe create ways to disrupt common patterns and scripts, to hop the conversation onto new tracks that are more productive?
Just because I think the viable solutions will differ dramatically (unless you're really into spiking coffees), so that affects the search space.
In most arguments you're looking at both the content of the argument (data + logic) and the person/ context providing that argument. The updating process might get derailed when the subject doesn't believe or trust the argument source.
It prob doesn't apply to trauma, but for strongly held beliefs that seem unshakeable in econ, politics etc, does seem to play a large role. I don't know what's trapped here, whether it's the prior or perhaps it's the updater that also considers the validity of the source of any new argument. This applies when it's still a neutral source since that would *still* require you to at least believe your ability to update is not somehow screwed up.
Another factor is that a lot of the time the evidence channel really is quite narrow.
I know someone who is convinced that vaccine side-effects are very dangerous and downplayed. I thought about trying to put together an actual analysis on the subject and presenting it with "I'm not asking you to trust 'them'; I'm asking you to trust me, because I actually checked" but then I realized I can't do it. I could probably get the post-approval surveillance data, but I don't have the skill to construct a demographically-matched control. Nor can I look at someone else's analysis and reliably tell that it was done correctly. Nor can I look at another analyst and reliably judge them regarding statistical skill and virtue of lightness.
Similarly, someone with a fear of dogs can find one easily, but what if you have a fear of sharks and the local aquarium doesn't have any?
So far so harmless, but how much US policy is written by people with a pathological fear of muslims who have never actually met one? Or who figure those chose to move to the United States are not representative of the ::handwaves at middle east::. That last point isn't actually wrong.
Is it possible to be so homophobic that even openly gay people refuse to out themselves to you? I bet it is.
I used GIMP to join the top half of the dark image with the bottom half of the light image, and the chess-piece pixels do in fact appear to be the same in the two images.
Nice article. I wonder if you could differentiate partisan results based on argument style — analytical arguments vs emotional appeals vs anecdotes. Logos vs Pathos vs Ethos.
Also, were there really accusations of dogwhistling on Biden's BLM support? I haven't seen that before, and a cursory search didn't find anything either.
I'm not convinced that trapped priors explain the capital punishment experiment. An alternative model: people's certainty about the topic reflects how well they think they can justify their opinion.
If you have a strong opinion you want to be able to talk at length about why that opinion is correct. If you barely have anything to say you look foolish.
A study that confirms your opinion is something you can cite in support, that therefore lets you get away with a stronger opinion. A study that disconfirms it is not as helpful, but it doesn't make you less able to support your opinion. Possibly you can still wring something useful out of it, but at worst it doesn't change anything.
In this model the subjective probability isn't affected by evidence, just the social expression.
Your comment about gradually increasing low level exposure getting around the strong prior made me think of the descriptions of radicalization. Eg you start with a strong prior against holocaust denial, so would reject a direct denial argument, but if you are slowly introduced to related arguments then in time you'll be willing to accept it.
Which raises the larger question of if these techniques are content neutral. Using the same technique as reducing fear of dogs could one be trained not to be afraid of being hit by cars, so walk in front of traffic?
Your discussion of left-wing and right-wing priors makes it all the more clear why it is so difficult to be a libertarian. To us, almost everyone we deal with is ludicrous. It is very frustrating having a political conversation with 95% of the public.
1. People should aim to have as few priors as they possibly can, and to chose them very carefully.
2. Priors can be dislodged in many ways, but the overarching theme is γνῶθι σεαυτόν (know thyself).
As I was reading the article my mind immediately went to PTSD before it was even mentioned. As someone who has had to do serious personal work because of PTSD, I was definitely dislodging a lot of trapped priors (about myself and the world), but I think one reason they were there is that they helped me keep the world much simpler. Dislodging a trapped prior (as someone who has had to do a lot of it) is really painful because it probably is naturally going to dislodge other beliefs and create a more complex picture of the world that will require a lot of work to ingest.
Re. the wine effect, I noticed many years ago that if I forgot what drink I'd poured into my cup, as I often do, that on drinking it I wouldn't say 'Oh, this is apple juice", but "Omigosh what is this strange taste that I've never tasted before?" Then, when I realized it was apple juice, the taste would suddenly shift to conform to apple-juice expectations.
I like to imagine that forgetting what's in my cup provides me with a "more-true" sense impression, but that desire is probably based on a bankrupt epistemology.
I think this is somewhat related. I went through a really really rough patch of trying to recall a traumatic event I had blocked out. I had told a loved one about it. A month or two later we tried talking about it again. We started off on normal conversation and then when I knew she was about to bring it up something happened. I felt like I lost my hearing. Like a mental discontinuity happened and no audio from her reached my ears. I only remember feeling like I was shriveling up on the inside. After her sentence I did what I could to quickly change the subject. This post reminded me of that. I feel like I have a better grasp on it now, but at that moment It literally seemed like I went deaf to stop hearing the words I wasn't ready to hear.
I wonder if there's any other cases of that happening out there.
It seams obvious to me (which means it is probably wrong) that the reason the prior doesn't update and the bandwidth on the sensation is throttled is because the person experiencing the situation has a flood of additional sensation that the context/prior weighting is generating.
The Rottweiler is scary. I am not only perceiving a dog wagging its tail, but also perceiving that my heart rate is accelerated, my muscles ache and want to pump, my hands are clammy, etc. If I can only pay attention to so many things at once, some of the immediacy of my physical reaction is going to take up my perception. Furthermore I'm now going to add an additional context on top of "dogs are scary" but also "when my heart races like this, it isn't a good thing" and "when I act scared my partner/parent is ashamed of me (which in shorthand is heard internally as: "I'm gathering shame right now")"
Exacerbating "bitch eating cracker syndrome", I think, is the escalation of moral investment in the designation "bitch." If you slap an ostensible bitch for eating a cracker, then you better be right that she's a bitch... otherwise you're a moral monster. In fact, you may need to up-level the designation to "evil bitch" in order to resolve your cognitive dissonance.
This is my hypothesis on how people turn abusive, or at least how my (ex) husband went down that path. In many respects, he is/was a very good man. I think he was trapped in the belief that I was an evil bitch, because what sort of monster physically abuses a non-evil human? This may be especially challenging for him because of his splitting - his world is full of angels and devils, with little in between - and thus he had a long way to fall. As trite as it sounds, I think the only hope for him is to learn to accept and forgive himself. But I'm not sure this is in the cards... may god have mercy on his soul.
Possible editing error: In the section "Trapped prior: the more complicated emotional version", you introduce van der Bergh's idea that negative emotions reduce the bandwidth on sensory input. Then in the section "Reiterating the cognitive vs. emotional distinction", you introduce this same idea again, phrased as if it were the first introduction.
What is the usual time frame of studies like this on the entrenchedness of priors? Just typical minding based on myself, I can definitely see and feel my skepticism tingling whenever I am presented with evidence that I am wrong, but in the long run the political stances I tend toward have changed quite a bit, say over the past 20 years. I don't think I ever particularly strong stances and I still don't, but in general temperament they have changed. I'm not claiming to be abnormally rational or anything. I think I just don't care much and find myself expressing agreement with whatever happens to be considered acceptable to my peers, and those peers have changed over the years.
On the capital punishment thing, what was the evidence these people were presented with? This seems like a question of values. Values are fundamental, not based on evidence, so I don't know what should be able to change them. This feels like a manifestation of the is/ought gap more than trapped priors. For questions of value, all you have are priors. If they ever update, it's by fiat or epiphany, not evidence.
I'm reminded of KGB defector Yuri Bezmenov's take on trying to reason with demoralized people; "you cannot change their mind, even if you expose them to authentic information". Scott enumerates some potential ways to escape trapped priors. But if I were to channel Bezmenov's criticism of the Soviet Union I then have to ask, "is it possible to take advantage of this belief model to better convince people to defend your position"? I assume so.
Reinforcement of priors is one approach. For example, I really want this person to be afraid of dogs. Therefore, I'm going to keep showing them pictures of scary looking dogs. This is more or less "I'm going to read about how bad the other political side is from the people who agree with me already." (Is it?) But under this model anything that can decrease the strength of the raw sensation is another approach to convincing people. But this seems to be a more defensive strategy. However, I'm having trouble thinking of how someone would craft a situation where this is the method of attack. Thoughts?
Wouldn't it be great to just be the nice dog instead and only be conditioned on whether an action results in a treat? Sounds like a pretty nice prior if you ask me
Remember JJ Gibson, 1950's, mechanism of perception and organ participation, which shows that perception is a compilation of the person's environment and how the person interacts with it.
Observation on the extremes of the dog whistle problem. Policies have lots of consequences, both benefits and costs/positives and negatives, potentially even to the same people.
When a pattern of consequences arises which (at least seems) like it was not stated in the "This is what the Law is About" section (haha) - but the pattern is either really positive or really negative for some group or individual - the question is posed, Did they do it this way on purpose, by accident, or what combination? Did "they" not realize this would happen, or was it what they were after all along and they just hid it? Or was it not the primary goal, but a desired objective that the architects were aware of? Individual lives have that, someone can legitimately not realize, and at an individual level the person can also have motives they are unaware of - and they can also have definite motives and feign innocence.
The problem is, looking at the complex of unintended consequences of a policy created by a group, and then choosing the "every supporter meant this and knew it" option.
From the position of someone affected by the unintended consequences, if something is negative enough and goes on long enough, there is a "they reasonably should have known by now" position, which makes it possible to stop extending any benefit of the doubt to the architects of the policy. It allows someone to attribute intent and therefore be justified in loathing.
From an education point of view, sometimes people really don't know and don't realize.
I have to stop writing this. Maybe more later. Very interesting post.
So, I work as a Clinical Psychologist in the UK, and have been recently treating anxiety, including phobias and OCD using standard CBT approaches.
In the case of a phobia, the thing that limits people’s raw experience is typically avoidance of the phobia stimuli. This temporarily relieves anxiety, and thus strongly negatively reinforces the avoidant behaviour (by making the scary thing go away) and of course reduces the experiential bandwidth that the “dog” is not scary, meaning that they only have their prior to make predictions about future dogs with.
I’ve never used flooding, but my understanding is that it actually *can* work, so long as the person doesn’t freak out and avoid the phobia stimuli. As we have to do therapy ethically, if a person simply doesn’t want to be locked in the room with the Rottweiler, we’ve gotta let them out. But if we let them out at a moment of high fear, we actually increase the fear conditioning and avoidance. In order for a person to become habituated and de-conditioned to the phobic stimuli and past avoidance, they need to stay with the stimuli for long enough that the experiential data demonstrates it is not dangerous *and* for their amygdala to calm down. This typically takes 20-30 minutes, and I explain all of this to patients at the start. We don’t use flooding because few people could tolerate it enough to consent, whereas most people can tolerate the cute puppy stage of systematic desensitisation which enables them to gradually face more scary phobic stimuli.
The issue though with phobias is that the person understands that their phobia is in some way silly or harmful, and is prepared to do systematic desensitisation to overcome it. My patient who believed she had to check her children were breathing multiple times per night or they might die *knew* on some level that her belief was irrational, and wanted to challenge it, but the trapped prior and fear and alleviation when she checked kept her negatively reinforcing the checking behaviour. Exposure and response prevention changed that, but it was because she ultimately wanted to change and simply needed the method and a good therapist to guide her.
My sense is that some political beliefs could be similar. If you have limited data on a group of people and fear them, you will likely avoid them, both negatively reinforcing your avoidance and also not gathering data which could disconfirm priors. If you actually got to know some Republicans, or some Muslims, you might not find them to be that bad, if you actually spent long enough with them, but your avoidance could prevent you from ever doing this.
Our confirmation biases could positively reinforce seeking information that makes us feel better about reading information that agrees with our beliefs - we all want to feel we are correct. Conversely, the cognitive dissonance, or discomfort we feel when presented with information that demonstrates to us that we are in error could cause us to avoid such information, negatively reinforcing the avoidance and also limiting our exposure to such evidence. It could also provoke a defensive behaviour, such as stating that the evidence is in some way “lies” or “fake news” as to truly accept the data as being correct involves some discomfort or potentially embarrassment at realising one was wrong.
Potentially, the cognitive dissonance or embarrassment is even stronger the more outlandish and bizarre ones beliefs are, which could explain why people would rather continue to believe e.g. Qanon conspiracy theories than face the emotional discomfort for *long enough* that the discomfort fades, enabling them to continue to absorb new experience without avoidance.
I think the avoidance, time required for fear or discomfort to reduce, and defensive reactions could all be reasons why humans can’t simply absorb additional data and thus more rationally update. I hope this is a useful addition to your own model. I appreciate that you will know the basics of this in psychiatry, but assume that you’ve not actually spent the time in the room with the patient and the dog to find out what makes it so hard for her 🙂
In short, yes, people *can* become more fervent believers after a failed prophecy, but in general what actually happens varies. The paper looks at a number of groups that had failed prophecies and some continue and become more fervent, others fall apart, etc. I'd suggest reading the whole thing, but I think the summary version is that it seems to depend on how the group's leaders react in the immediate aftermath of the prophecy.
It's worthwhile to spell out why confirmation bias isn't actually rational: it's a failure to draw distinctions between the evidence itself, and your interpretation of it.
If you know you have a phobia of dogs, then, when you see a dog as scary, that shouldn't give you any new evidence that dogs are a threat. You knew in advance that would happen! It would happen no matter how harmless the dog actually was. The problem is that you ignore the layers of interpretation, and try to update as if you had encountered an objectively dangerous dog. If that had happened, it would indeed be evidence that many dogs are dangerous.
If you have a prior that vaccines are safe, and you hear about someone who died soon after being vaccinated, you will say that it was most likely a coincidence. This is a correct conclusion. Then, since this death was a coincidence, you will not update your priors et all. This part is wrong! The event is still a valid bit of evidence. It is something that would be more likely to happen if the vaccine were dangerous than if it were safe. You need to update (a tiny bit, in this case) in favor of "dangerous" - even though you are correctly confident that there was no connection.
Updating based on evidence, and the interpretation of said evidence, are two different processes that need to be kept completely separate. This is something humans are very bad at doing, hence "confirmation bias".
I'm not totally convinced by the move from subconscious trapped priors (snap judgments, sense data pattern recognition, phobias, PTSD, etc.) and conscious trapped priors (bias, beliefs, ideologies, etc.). The former is driven by raw associations and experience, but the latter arises from more sophisticated psychological constructs: identity, temperament, tribal affiliation. Rewiring the former seems doable and in some cases therapeutically useful, rewiring the latter seems both difficult and also perhaps not terribly desirable.
Is it truly wise in starting the post by talking about priors in the context of democrats and republicans? I understand that the distorted thinking people have around politics is a big problem, but that same distorted thinking also makes it even harder for you to reach anyone suffering from it (I.E. a large fraction of the population) when you start out the gate with "but what about reasonable republican arguments." They'd get angry and say that it's entirely justified for them to be aggressively opposed to Republican policies, and list any number of reasons why giving the benefit of the doubt to Republicans is like giving it to Nazis. I actively try to avoid news articles most of the time and I've read and enjoyed several hundreds of your posts, and it would be basically effortless for ME to make that list. It's an urge that has to be held back, an urge that as you say, seems reasonable to a person from the inside. And most people can't hold it back. So why start an important article like that, and lose a fraction of the audience from the start?
I mean, I understand the other failure mode. Give all uncontroversial examples and people will nod their heads along to your point and then as soon as something controversial comes up they won't apply the lesson to that one at all, act like it's a different case or just inapplicable. That's a tradeoff I suppose. But what does the side you chose on this particular tradeoff benefit you? it seems like democrats vs republicans was an example you picked because it was easy to reach for, not one chosen to be particularly illustrative. You don't do anything with that paragraph that could not have also been done by substituting in any other ideologies, and almost any other ideologies would be less rage invoking (less prior bending?). I don't feel like slipping the democrats vs republicans thing in at the start does anything meaningful to break political tribalism, but it will negatively tint the perspective of everyone who is not already extremely open minded.
(Oh, or maybe this is the kind of thing you're doing because your actual goal is to try to win over more republicans to your line of thinking, in which case perhaps it's a more useful paragraph than I'd thought)
I guess that element of the banner image is about updating priors instead of the Astral triad of Yesod, Hod, and Netzach after all. Alas, I suppose it was just a coincidence.
>I've sort of lazily written as if there's a "point of no return" - priors can update normally until they reach a certain strength, and after that they're trapped and can't update anymore. Probably this isn't true. Probably they just become trapped relative to the amount of evidence an ordinary person is likely to experience. Given immense, overwhelming evidence, the evidence could still drown out the prior and cause an update. But it would have to be really big.
>Sloman and Fernbach might be the political bias version of this phenomenon. They ask partisans their opinions on various issues, and as usual find strong partisan biases. Then they asked them to do various things. The only task that moderated partisan extremism was to give a precise mechanistic explanation of how their preferred policy should help - for example, describing in detail the mechanism by which sanctions on Iran would make its nuclear program go better or worse.
My take on this is that prompting someone for the mechanistic explanation confirms both that that 1) the belief directly serves its holder in terms of expected evidence and 2) there is a *specific* expectation whose negation might be evidence against the belief. When the context is generating predictions rather than commenting on someone else's work, it skews the utility away from belief-as-attire and removes the feedback loops of interpersonal politics that would drive people to extremes.
(Maybe there's also an instinctual element of leaving a line of retreat that become easier with moderated beliefs, but I'm not sure if that's strictly necessary for the effect. )
One issue is that your model may be too simple - in particular, there's more than two inputs.
Phobias are fairly simple - someone has an irrational fear of some stimulus. It makes no logical sense, they may even recognize that it makes no sense, but they still struggle with it.
But when we talk about, say, scientific knowledge, one issue is likely that people become falsely overconfident in their scientific beliefs because if they are often right about *other* scientific things, that makes it more likely that they're right about *this* thing. That is to say, they think that they are "smart science people", they are usually right about most scientific things, so when there's some scientific thing that's controversial, they side with whatever it is that their political beliefs tell them to, regardless of the actual evidence, but because of their science knowledge, they falsely end up believing that their beliefs are correct because they have other science knowledge that is correct, and that incorrectly gets carried over into the biased domain of belief.
This happens with a lot of experts in general, really; people assume they are good at X and therefore know better about Y, even though, in reality, Y is primarily a matter of knowledge and so being good at X isn't actually all that relevant.
This makes it inordinately difficult to change their scientific belief because their brain, rather than correctly applying it back as "you don't understand this thing in particular", ends up trying to apply it back against "you don't understand this thing IN GENERAL", at which point the counterevidence of "I know all this stuff!" comes back in and thus greatly devalues or even reverses the signal.
Someone who isn't so sure about science is going to have an easier time changing their opinion about some of these things if they aren't capable of cognitively isolating the "good at science" thing, becuase they don't think that they ARE good at science, and so their priors are more easily changed because they don't have the same resistance to changing their beliefs.
I think that people who can overcome this are the people whose view of science is "I am smart about science, so questioning this thing that I/people/the public/my tribe assumed to be true means that I am good at science". This causes the negative reinforcement loop to instead become a positive reinforcement loop, as it means that questioning common assumptions means you are *good at science*.
Of course, there is a downside to this, which is that if you combine this viewpoint with being not actually very good at science, you can end up believing stupid things or else having overly low confidence about everything, because you aren't very good at filtering out bad information.
Honestly, I sometimes wonder if this is why we have so many dumb people about some of these things - a lot of the antivaxxers I've met IRL are not vehmently anti-vax, they just are distrustful of them because they know enough to be suspicious of medical studies but not enough to actually be able to apply skepticism correctly to data. They're skeptical of everything, and would, ironically, be better off with a worse basic herustic, because they aren't actually good enough at "all things should be questioned" to actually apply it usefully.
The people who trust respected authority figures like Dr. Fauci are better off than they are.
Meanwhile, the people who are *actually* good at this stuff were saying to wear masks before the CDC said to wear masks. My mom bought respirators back in January 2020.
This may be why these sorts of things are actually so common - because having fixed priors is bad, but being incapable of actually learning anything is worse. Someone with relaxed priors who is actually good at this stuff will probably only end up with a few weird beliefs, if any, but most people are better off with the worse heuristic because they're not capable of using the better one.
This would also suggest that using psychedelics is probably a bad idea - permanently trapping people's brains open is going to probably lead to net negative consequences in most cases, as while there is a greater optimum to "open minded + competent" than "closed minded + competent", "open minded + incompetent" is probably worse than either.
Which is probably why most people are closed minded rather than open-minded, but there's enough people who benefit from being open minded that the trait hasn't been removed from the population.
I have pondered this phenomenon as a layperson for quite some time; my intuitive thought was that at some point the Bayesian prior is so strongly believed that contrary evidence is taken as an indication that the source of the contrary evidence is unreliable, not that the prior itself might be wrong. I'm not good enough at the math to model that, but any given observation in the Bayesian model doesn't have to be viewed as either true or false; it (like anything else) is assigned a likelihood. But, intuitively, if the NY Times reported (seriously) that the earth was flat, I would not decide the earth was flat, I'd start doubting the NY Times (more than I do; lolz).
FWIW my internal shorthand for this phenomenon is not "trapped priors," which is fine, but rather getting stuck in a "Bayesian Black Hole." If you like the "Bayesian Black Hole" framing, feel free to use it - no attribution needed!
Is it the case that when we're young, our minds run sensation and context through a sensation-weighted algorithm; and once we're older the algorithm switches and weighs context more heavily? Is the context vs sensation-weighted algo thing the same as low vs high neuroplasticity?
You observe that, even with a very strong prior, a "reasonable" update function should be able to notice that the experience was at least marginally less bad than predicted, and slowly update away from the prior. But that empirically, this often doesn't seem to happen.
I feel I can very easily imagine a minor and subtle defect in the update function that would cause this failure. For example, suppose evolution "wants" you to update 20% more rapidly on negative stimuli (for similar reasons to why we have loss aversion). The way that *ought* to be implemented is: calculate how far your beliefs would move "normally", then multiply the movement by 1.2 (this will always cause you to move in the same direction as before, just farther). But suppose the *actual* implementation is: multiply the "badness" of the perception by 1.2, and then update on that. If your prior was approximately neutral, this works out to nearly the same thing. If your prior was bad, but the "perception" (weighted combination of sensation and context) was good, then the multiplier isn't used and you update normally. BUT if your prior was bad, AND the perception was only *slightly* less bad, then multiplying it by 1.2 will make it *worse* than your prior, and cause you to update in the wrong *direction*.
That second implementation isn't *defensible*--you couldn't rationally argue it is somehow preferable to the first implementation. But I can easily imagine a lazy programmer doing it without realizing the issue, checking that it gives expected results in some basic tests, and then calling it done.
Evolution is certainly not MORE foresighted than a lazy programmer.
But even that explanation might be overcomplicated. When I read your description of the predictive coding theory of the brain (probably this was in your review of Surfing Uncertainty, though I'm not certain), one detail that REALLY REALLY jumped out at me was when you said the brain compares the sensory input (with error bars) to its prediction (with error bars), and if they match, then it sends THE PREDICTION (not the sensory input!) up to the next-higher layer of abstraction for further processing.
That was just your summary; I don't know how much the evidence supports that detail, or even if you were consciously making the distinction when you wrote it. But if that gray box in your diagrams isn't so much taking a weighted average of the inputs as using some threshold function to decide which input to IGNORE, then that would compactly explain the failure to update in the direction of the sense input.
(Disclosure: I immediately thought that this detail would make sense of a lot of arguments over board game rules where both sides are convinced the rulebook supports their own position, so maybe I was predisposed to believe it.)
The shortcoming of such an approach is that so many - and arguably the worst - biases are not based on direct experiential "priors" at all, but on things like values and sensibilities. So even if one found a way to help people prioritize new "raw" experience, would it even touch the problem? This of course particularly applies to politics.
Having said that, the research on psychedelics does suggest the more "raw-like" experiences it facilitates leads to a coalescence of a sort around more relational values. Though the quality of these studies hasn't been that great, and likely many went into the experience with a particular openness to being changed.
An openness that many if not most of those with "trapped priors" or rigid values are unlikely to share.
If one is aware of trapped priors and how they functions (such as you've described), can that help make us less susceptible to them? Not necessarily for traumatic things, like the "scary dog", but the more mundane, like political bias and polar bears?
A lot of my strongly partisan/conspiracy theory-ish beliefs are things I just kind of choose to believe because it feels good to believe them. But I'm capable of setting them aside in favor of more reasonable beliefs when I need to. It's not really trapped priors or any kind of issue with rational thinking. I'm just choosing in these moments not to think rationally.
"(My usual metaphor is "if God came down from the heavens and told you ... - but God coming down from the heavens and telling you anything probably makes apocalypse cultism more probable, not less.)"
What if God came down from the heavens and told you He doesn't exist?
Interestingly, you see a similar sort of sensitization and desensitization with allergic reactions. After an initial sensitization to certain allergens (from e.g. insect stings), subsequent allergic reactions can be significantly worse. Despite that, allergy shots can also work for those same allergens, with the main difference being the amount of the allergen you're exposed to at each time - gradually increasing as you become desensitized to it.
It makes me wonder if the underlying mechanism is similar in each case.
The supplementary materials for the Sloman and Fernbach paper include the prompt they used to ask for mechanistic explanations:
“Now, we'd like to probe your knowledge in a little more detail on two of the political issues. This is the first one. Please describe all the details you know about [the impact of instituting a 'cap and trade' system for carbon emissions], going from the first step to the last, and providing the causal connection between the steps. That is, your explanation should state precisely how each step causes the next step in one continuous chain from start to finish. In other words, try to tell as complete a story as you can, with no gaps. Please take your time, as we expect your best explanation.”
So I may have already posted a comment here, but it's not appearing.
This is brilliant.
My intuitive sense is that we assign probabilities not just to priors, but to observations, including the likelihood of the source. I generally believe the NY Times, but if they reported that the earth was flat, I'd lower my estimation of them, not decide that the earth wasn't round. In this way a sufficiently strong prior can overwhelm contrary evidence by degrading belief in the observation.
I call this a "Bayesian Black Hole." If you like the term, feel free to use it.
Formal debate training, where one must research a proposition and then be prepared to argue persuasively either side, was once a common requirement in education. It helped foster critical thinking. Similarly, the Catholic Church adopted the concept of a “Devil’s Advocate” to argue against the elevation of a person to Sainthood. I have found, and there is much written about the technique, that formally designating a “Devil’s Advocate” helps break down biases in decision making. It helps eliminate the “you can’t talk about that” factor.
Van der Bergh et al see what they are studying as "Better Safe Than Sorry" information-processing strategies. Carrying that over to trapped priors the question is why it would be unsafe to release the prior and allow it to be updated. What safety is preserved by keeping that prior trapped?
As a coach, I have consistent success asking people what's at risk if they were to reconsider certain deeply held beliefs, and asking if there are better ways to mitigate or tolerate that risk than the strategies they've been pursuing. (Scott's Mental Mountains post nods at similar interventions.)
This only works if the client is suffering in some way because of their held prior, and has reason to question it. But it could work for rationalists as well: asking "What's at risk for me if I were to change this belief?" may draw our attention to where our own reasoning might be motivated.
This is a great write up, but I think it overlooks the significance of the degree to which this phenomenon exists among the public. It's easy to see how this can occur in political partisans and people with phobias, but what is rarely discussed is how powerful the phenomenon is on culture war topics, even with otherwise extremely rational people. For example: Hacker News and /r/SlateStarCodex are two communities with very high concentrations of rational people. However, if certain topics come up, individuals in these communities almost without exception are rarely able to keep it together and engage in a logically and epistemically sound conversation, free of rhetoric. It seems that people are simply unable to practice the Socratic method on certain topics.
I believe here lies an excellent topic for some research. Typically, studies tend to focus on average person, who usually know they are involved in a study. I think it would be interesting if a way could be found to seed deliberately controversial articles directly into such communities, and then have researchers observe how these rational people behave irrationally (including things like mind reading and predicting the future), and perhaps even intervening to see if any way(s) can be found to break them out of the illusion. And even if that process was a failure, then see what the reaction is when they are let in on the fact that they were the subjects of a study, and shown a copy of their comments, with logical criticisms noted along side each comment. If this could be done by someone who the community members respect, I think it could be very informative, and possibly lead to some form of a saleable approach.
"In the old days, psychologists would treat phobia by flooding patients with the phobic object. Got cynophobia? We'll stick you in a room with a giant Rottweiler, lock the door, and by the time you come out maybe you won't be afraid of dogs anymore. Sound barbaric? Maybe so, but more important it didn't really work. You could spend all day in the room with the Rottweiler, the Rottweiler could fall asleep or lick your face or do something else that should have been sufficient to convince you it wasn't scary, and by the time you got out you'd be even more afraid of dogs than when you went in."
This sounds odd; why would they apply a treatment that didn't work? Furthermore, when I looked it up, a lot of pages claimed that the treatment did work, with some even claiming that it was highly effective.
If you're interested in how doomsday cults respond to the lack of doomsday, Stephen O'Leary (at USC until he died last year) had research and a few books about that. He had put a group together to watch the world not end in the year 2000. I remember reading some of his [i]Arguing the Apocalypse: A Theory of Millennial Rhetoric[/i] at the time, but I never followed up after the world failed to end.
There's some evidence that, when someone has a relatively strong position in favor of X, ridiculous over-the-top propaganda from pro-X extremists can be more effective in making the pro-X belief less strong, compared to a reasonable argument or well-designed study refuting X. I remember reading about a study of attitudes about Israel/Palestine that came to that conclusion: people who supported Israel were made more extreme in their position after viewing a reasoned argument in favor of an independent Palestinian state, but they became more moderate after viewing over-the-top crush-the-Palestinians propaganda. Unfortunately I can't find the study now.
I know that for me personally, there are many cases where extremism from my own political camp was more effective in moderating my political beliefs, compared to arguments from the "other side."
Maybe this works because, since the content at first glance appears to confirm your beliefs, you don't reflexively decrease the weighting on the sensation compared to priors? I would suspect that this only works if you believe the extreme ridiculous pro-X propaganda to be genuine; if you think it's actually a straw man from the anti-X side, your pro-X prior will only be reinforced.
Scott, I have some articles on priors that are relevant to this post, but cannot find the email to reach you at. Please let me know which is best so I could send them to you!
I think part of the story that might be missing is that your priors can feed back and influence what sensations you even collect. Consider the fisherman who never casts a line into a certain part of the river because he "knows" that the fish never bite there and it would just be a waste of time. I wonder if human attention is gappy enough that positive interactions with a dog are literally invisible to the cynophobe.
What's going on with psychedelics in regard to priors seems pretty straightforward: they potentiate Bayesian updating.
It just turns out sometimes that's truth-directed, and sometimes it's not. This inconsistency of effect is not because of anything about psychedelics. Rather, it's simply because Bayesian updating isn't an oracle for truth.
Rather than seeking a "pro-rationality" intervention, one should seek practices of good mental hygiene, including some of those Scott mentions. The effective model is more like pursuing good diet and exercise, and less like debugging code.
This is a similar situation for NDE’s. People die and ‘come back’ different. Why? Their dying causes a release of trapped biases as to who they are supposed to be. A break in the narrative. Death exposes the absurdity of our notions about life and makes obsolete the fears that previously had such a hold over us, because we have now experienced the ultimate ‘happening’ of dying that what we previously feared seem absurd.
This is one of many reasons why we should be using the word ‘Steelman’ as much as possible. When it comes to trapped priors in the context of personal beliefs, political or otherwise, having to steelman the argument of the person you disagree with will probably help in reducing the strength of your priors (assuming they are actually capable of granting charity to the opposition). I’m so sick of people assuming the worst of members of the outgroup.
This thing—The Trapped Prior—is exactly the concept I've been looking for. This brings together a chunk of old rationalist material, and arguably The Cowpox of Doubt (https://slatestarcodex.com/2014/04/15/the-cowpox-of-doubt/) discussed the negative version of trapped priors, i.e. psychological inoculation. After hearing a large number of poor arguments for P, you will evaluate even objectively strong arguments for P is poor. This is it: The Trapped Prior has afflicted you. Your own thinking cannot recognize it; you will feel like "I evaluated this argument fairly, as I evaluate all arguments, and it's clearly a bad argument."
Another interesting line of thought is whether or not the Ethnic Tension-style arguments (https://slatestarcodex.com/2014/11/04/ethnic-tension-and-meaningless-arguments/) somehow act on people's ability to evaluate policy proposals by creating a context such that policy proposals for a certain position will get evaluated more positively.
Another possible method to explore is to pretend to believe in an opposing prior. Like a democrat could pretend to be a republican, or a cynophobic person could pretend to love dogs. I imagine a playful, humorous setting where you start out with a ludicrous parody of the position opposed to the one you're stuck in and end it with laughing about it. Then maybe extend it for longer periods of time or try to make it more "realistic". And only after you have some familiarity with the role, bring in the experience and try to look at it the way this role you're playing would.
You're assuming exactly two inputs per evaluation to the update function, and also that the function is itself immutable. Do you trust those assumptions? Should you?
On the subject of overcoming phobias there was a UK TV hypnotist/NLP practitioner called Paul McKenna who claimed to be able to cure people's phobias within half an hour or so. You can look him up on YouTube. He is not the most charismatic person but I do remember watching a person who went literally white in the face seeing a dog a hundred yards away and within the next hour they had been transformed into someone happy to have a dog leap up and lick their face. It was quite remarkable and I don't think it was a fake.
Scott said - ....self-serving bias. People are more likely to believe ideas that would benefit them if true ....... I suspect that these are honest beliefs
Elephant in the brain (Simler/ Hanson) explanation. It is an Darwinian advantage to sincerely believe one's own lies, so we've evolved to be really good at it.
On the therapeutic side, it seems like being in a room with a rottweiler while having cynophobia might be unpleasant enough that "dogs cause unpleasantness" gets updated by the experience... but naive mechanistic reasoning suggests that ameliorating contextual factors (like body armor, or the dog being chained up, and intervening protective glass, or also eating ice cream, or all of these factors at the same time) could modulate the process.
This feels consistent with a strongly felt intuition of mine that a FUNDAMENTAL pre-condition for learning, and thinking creatively, in general, is a (basically justified) feeling of "safety".
The most effective protection I can personally imagine is simply "a adequately powerful agent close enough to ensure my safety". Also, I think it might be common for people to have phobias for things a parent encouraged them to be afraid of as children (roughly) because the parent thought the fear would be more helpful than harmful?
When I google the plural with ["trapped priorS" attachment style] I find nothing. Modifying to search for the singular form gives a tiny number of ecology papers about wild animals being "trapped" "prior" to some measurement or treatment that the research paper is focusing on. I'm wondering if "trapped priors" is your own name for this? Are there other keywords for this that might turn up other veins of related research?
Related to parental judgement that a phobia might be good to install... I've talked with people who were phobic about various critters, and they basically have never been interested in curing their phobia. It is like they "want to find something bad tasting" because then they'll eat less of it, which would be good if it is objectively bad to eat? Except they are reasoning this way about fear and avoidance instead of taste and food.
This helps me make sense of why "talking through the mechanistic steps" around a big vague fear might cause people to become more mentally flexible: maybe visualizing more intervening details allows them to imagine a different action than "shunning the first step" which could ALSO protect them from the very last domino falling over in a way that they think will cause objective harm?
If they thought the fear was their only good strategy for avoiding harm, they might cling to the fear.
I think self serving bias along with the rich & powerful having leverage to change the political / economic system is enough to explain cost disease.
The longer everything exists in some kind of stable state the more people with power figure out ways to corrupt the markets & bureaucracy to their benefit. Citizens United, Surveillance Ad Tech, Corrupt Unions, CEOs making 300% more than workers. There won't be a single explanation because it's simply processes being corrupted by people with power who likely are deceiving themselves that this is a good thing.
"It can happen (in theory) even in someone who doesn't feel emotions at all. If you gather sufficient evidence that there are no polar bears near you, and your algorithm for combining prior with new experience is just a little off, then you can end up rejecting all apparent evidence of polar bears as fake, and trapping your anti-polar-bear prior."
I don't see how this can be true. If you're very sure there aren't polar bears, and then you keep seeing bits of white fur and tracks that look like they could be from a big bear, you might not notice them at all, or you might be confident they're fake, or that the fur is from a dog. But none of that can make you MORE confident that there are no polar bears.
The only way evidence of polar bears could make you more certain there are no polar bears is if it convinces you that, say, someone is out to get you by faking signs of polar bears to make you look crazy. In other words, an emotional situation.
Comparing phobias and political/cognitive biases is an interesting exercise. People with phobias may recognize they have an abnormal condition and can seek or accept help. People with unassailable political convictions often don't recognize their state as a problem, even when their beliefs are relatively fringe and impose real costs like the lost of family and friend relationships (QAnon, cults).
The difference is obviously in the nature of sensory evidence: after all, one can see dogs being walked everywhere by unafraid people, but it's impossible for a bystander to directly experience an unjust death penalty. Generally, the only way to absolutely know a person is innocent is to be that person.
I'd theorize that in cases of abstract/intellectual issues like politics the "raw experience" channel is fed by the imagination, not the senses, and the imagination is essentially just a remix of context priors, plus whatever one is trying to imagine. So context in => context out and beliefs won't move away from context
Then why would argument ever work on anyone? Presumably when one can imagine their conversation partner's specific argument more easily, allowing a path for non-prior concepts to enter the process. This naturally works better when the incoming concepts more closely agree with one's priors, or possibly for individuals that naturally give their priors low weights.
I'm particularly tempted to think that a link between low prior weights and a flexible imagination might also correlate with some people's ability to think much more abstractly than others.
You're conflating two things with very different mechanisms of formation because they both present similarly as "trapped priors". Cynophobia (or many traumas really) can result from a single bad experience and is generally more context specific. It's the result of one-shot learning. A rat learns the association between the bell and the shock in as little as one trial: the limbic system at work.
Political biases are the result of much more evidence accumulated over a much longer period of time, are far more context-independent, and are the trigger *to* intense emotion rather than being triggered *by* intense emotion. A much better model would be that of habit formation. The mechanisms of learning/unlearning are far less plastic in this case, responses are far more reflexive and the emotional component is secondary.
Just for fun I made a quick spreadsheet to simulate what would happen if your priors updated slowly.
Say you're a perfect Bayesian agent embodying Bayes' theorem. You start out very certain that all dogs are scary, and it's very unlikely that happy puppies even exist, but then you see an obviously very nice puppy. You'll update immediately from 99.9...% confidence that dogs are scary, to something like 1% odds that all dogs are scary.
But if you modify the update such that the experience of seeing a happy puppy only updates your prior half as much as much as it should, it takes you ~4x as many exposures to the happy puppy before you reach the "ideal Bayesian agent" confidence that happy puppies exist.
And if your prior only moves 10% as much as it should, it takes you *~25x* as many exposures.
Given my old days in a neuroscience lab, I would conjecture that a trapped prior can be facilitated by long-term potentiation and long-term depression (https://en.wikipedia.org/wiki/Long-term_potentiation) in whatever synaptic connections that are involved, creating a lasting link that is resistant to changes in stimuli. If so, then a more promising way to change the behavior would be the pharmaceutical or therapeutic interventions that reverse LTP/LTD in specific brain regions, like maybe an NMDA receptor blocker (memantine?), and whatever EMDR does, possibly. And then we would finally get a handle on the addiction to unhealthy Vietnamese noodles, and other pho-bias.
The “if God came down from the heavens and told you” scenario plays out amusingly in “The Oven of Akhnai”, a story in the Talmud.
To paraphrase, two rabbis are arguing over whether a new type of oven is kosher. Eventually Rabbi Eliezer gets frustrated that his arguments are failing to persuade Rabbi Joshua, and he declares, “Look, if my interpretation of the Law is correct, this carob tree will move.” At once the tree leaps a couple of yards to the left and re-roots itself. But Rabbi Joshua just replies to the effect that strange trees hopping about is no basis for a system of determining ritual purity.
So Eliezer says, “Well then, if I’m right, the river will reverse it’s course!” And the river promptly turns and flows toward its source. But Joshua remains unimpressed.
“OK,” says Eliezer, “If I’m right, the walls of the yeshiva will tumble to the ground!” And they do start to fall - until Joshua scolds them for getting involved in a halakhic dispute without proper training. (The walls then hedge their bets by standing up again, but not quite perpendicular.)
Finally, Eliezer has had enough. “If I am right,” he cries, “the very Heavens will declare it!” And from the sky an immense voice booms out, “ELIEZER’S RIGHT, ALRIGHT? JUST GIVE IT A REST ALREADY.”
But Joshua replies simply, “Torah is not in Heaven.”
You’d think that this extraordinary assertion of the primacy of evidence over the highest conceivable authority might have licensed an early flowering of scientific reasoning in Jewish culture... but sadly the idea doesn’t seem to have been extended much beyond, “We should bicker incessantly about liturgical texts.”
Simulated annealing is a simulation of real annealing, which is a metallurgical technique. After beating a piece of metal for a while, some of the atoms in the original piece get dislocated. This means atoms actually get pushed around and crammed into open spaces in the crystal lattice. This makes the piece slightly denser and also more brittle.
To overcome the brittleness, the piece is briefly reheated. This causes some of the connections around the dislocations to break and new ones to form. This makes the piece less brittle while also preserving the atoms-forced-into-nooks-and-crannies density that came from working the piece. The final piece is denser than the original yet not as brittle as it was before annealing.
Simulated annealing is a technique in computing which is designed to escape local extremes an optimization algorithm might get trapped in. It takes lots of forms, but a common one is to speculatively "teleport" your current position in the space you are optimizing to another semi-random location. This briefly makes the optimization worse, in the hopes that further optimization from this new location will yield a better global result than the previous optimum. This is analogous to reheating the metal and allowing it to recool, where the temperature of the piece functions the same as the cost of the optimizing cost function.
Annealing is both contexts is a process of making a crystallized structure more plastic in the hopes that it recrystallizes into a better form than the initial.
I'm an engineer and not a doctor/psychologist/etc so wouldn't know a thing, but the theory kind of makes sense and your post made me remember that, the underlying mechanics seem similar.
I tried to convince my girlfriend to go there for her extreme dog phobia, but even the thought of the therapy triggers her trapped priors so we couldn't try that yet. That's kind of a recursive trapception we need to take ourselves out from. One last detail is, when she's really drunk she loses the phobia (and pets dogs), but it comes back when she's back sober.
Seems to me you are kinda confusing two different models. On the emotional model we are imagining an agent who engages in some kind of reward based learning. If you allow a feedback from the model to the learning function you can easily see how it would end up in an state where objectively ok experiences were actually interpreted negatively.
On the Bayesian rational updating model there shouldn't be any means for a high prior to trap your beliefs. And that's not just an assumption of perfectness but seems like you'd have to intrinsically build in this kind of bad behavior.
The problem is simply that the more certain you are that *all* dogs are horrible and dangerous the higher your prior should be that the dog will attack you and thus the greater the evidence updating will give when the dog doesn't bite. You can't explain why the individual would keep predicting a high chance of dog attacks even though they don't happen.
I suspect, that if you actually ask people it's not that they will report crazy high priors for any concrete idea. Thus, I think the take away should be that this case is better explained by the emotional reward idea first pretty well and the pure Bayesian prediction model (unless you are predicting emotional reaction in which case you just grated on the reward model) not so well.
I don't think psychedelics are necessary to assist in changing priors more rapidly, but really any mind-altering substance that induces a pleasant emotional affect. MDMA would likely work much faster and more effectively than psychedelics. Putting someone who is afraid of a dog in a room with a dog is just as likely to terrify them as cure them, but put them in a room with a dog on MDMA and they would probably not be able to resist cuddling or at least would have profoundly reduced fear.
But even MDMA isn't necessary, alcohol works too. Alcohol works quite well for overcoming all kinds of mild phobias and social anxieties and opening one's mind to things one is generally against.
Another things that works pretty well is being under the influence of whatever neurochemicals are coursing through one's brain when they're in love. I've persuaded several exes out of life-long strongly held religious and political positions while they were in love with me. Falling in love or maybe even lust or strong like with someone who holds convictions you disagree with is a strong motivator to becoming open to changing them.
In general though, your post (which I liked a lot) was somewhat depressing, as someone who likes to take my dogs hiking, I am always trying to find a resolution for people who are irrationally afraid of dogs. Because I don't like being screamed at by someone who is terrified, as has happened. One of my dogs looks like a big black wolf, though he has the most bite-inhibition I've ever seen on a dog...even as a puppy he refused to put his teeth on skin...and poses essentially zero risk of ever biting a human, but he looks scary to people afraid of dogs.
I have always wondered how it is that so many people who fear dogs claim to have been "attacked" multiple times, which is statistically extremely unlikely for anyone who doesn't make a career of working with them, and I've suspected that what they consider an "attack" is exaggerated -- i.e. a dog running towards them or barking or making noise. Your post indicates that is likely the case.
Though it is even worse than just a matter of trapped priors, however, because people with this phobia often behave in ways that make their priors much more likely to be confirmed. People afraid of dogs behave in a manner that dogs interpret as "sketchy person who is acting weird and is a threat" and dogs then become more likely to actually growl at or menace the person. This certainly is the case in human social contexts. If someone has a prior that "X category of people discriminate and look down on people like me", they are likely to behave defensively and in an unfriendly manner -- i.e. not making eye contact, scowling, using a hostile tone -- which then increases the likelihood that they get a like response, which then confirms their prior. Hence the chip on one's shoulder ends up being realized.
What a terrible cycle to get out of between competing political groups. One's priors leads one to assume the other person is going to lie or be an idiot, and so do theirs, you both interpret whatever the other says through that filter, AND you both actually behave in a ruder and more hostile way than you normally would, thus lending both assumed and actual validation to the other's priors. What a shame that nowadays, it is fashionable to preface one's argument with "speaking as an XYZ identifying person, I think that blah blah blah", which just sets up everyone's brain to trigger off the identification rather than the argument.
Great article, I’d suggest an alternative mental model that solves this problem:
> In most reasonable weighting functions, even a strong prior on scary dogs plus any evidence of a friendly dog should be able to make the perception slightly less scary than the prior, and iterated over a long enough chain this should update the prior towards dog friendliness. I don’t know why this doesn’t happen in real life, beyond a general sense that whatever weighting function we use isn’t perfectly Bayesian and doesn’t fit in the class I would call “reasonable”. I realize this is a weakness of this model and something that needs further study.
In my model, the brain’s sensory processing unit is a prediction machine. It receives raw experience, guesses how the higher units of your brain will feel about this input, and then processes the input to highlight relevant details based on its guess. If it’s right about what it guessed, the processing is reinforced. If its wrong, then it adjusts in the appropriate direction.
When you are trapped in the room with the Rottweiller, it guesses you will be terrified, and will want to highlight the parts of the experience that are potentially dangerous. Since you *are* indeed terrified, it guessed correctly, and the processing is reinforced.
However, when you are gradually introduced to puppies, your processing guesses wrong. The higher units of your brain are not terrified, and so the signal is sent down to the sensory processing unit to change its predictions for next time.
Direct persuasion is hard, almost downright impossible for strongly-held beliefs. That's because humans are social creatures, and our lizard brains are very attuned to the social hierarchies in our world around us.
If my friend Bob tries to convince me of something in direct contrast to my political beliefs, yes, maybe my political priors should be updated based on the strength of the argument and based on my perceptions of Bob's trustworthiness and intelligence, but at the same time my priors about Bob are being updated, too. And because we evolved as social animals, we are much readier to shift our perceptions of people.
And there's the fact that if I do change my belief, that also sends a social signal back to Bob, and he's going to update his priors about me.
The medium is the message, and when that medium is another person, interpersonal dynamics take over.
If somebody came along and advertised a "prior-unsticking therapy for improved rationality," the first thing I'd want to know isn't how they unstick my priors, but how they determine what I'm being irrational about.
And also maybe the opinions people tend to hold after embarking on this therapy, so that I can judge whether I want to hold those opinions myself.
Right now, we have deemed some opinions to be ethically medicalizable: personal delusions ("I am Napoleon!"), paranoia ("everyone's out to get me"), and others. Insofar as prior-unsticking is a therapy for such conditions, it seems unproblematic.
But if we're considering roping in opinions that are wacky, but not currently medicalizable, then I'm concerned. Even if everybody has a tendency to be irrational about politics, that doesn't mean that everybody (or anybody) should be subjected to prior-unsticking therapy over their politics.
Political irrationality eats everything, and it will eat this too. Just wait for the day when a decrease in support for an unpopular opinion is deemed as evidence for the efficacy of prior-unsticking therapy...
> I think van der Bergh argues that when something is so scary or hated that it's aversive to have to perceive it directly, your mind decreases bandwidth on the raw experience channel relative to the prior channel so that you avoid the negative stimulus.
How can this be true when e.g. meditation techniques tell you to focus on the actual sensations of the pain, which I find are less aversive once I force my experience channel wider? Pain goes from "ahh the pain" to "a sort of weaving blooming sensation in my abdomen, with unpleasant valence". It seems like explanations of the form 'helps you avoid negative stimulus' can't be right due to how often context makes things worse. I would faintly bet on it making thing worse on net – it feels like there's no experience so good that it can't be ruined by multiple bad experiences (ordering a dish I strongly expect to be good each time, that is slightly less good than it was last time, until I get sick of it), but some things feel so fixedly bad would take enormous effort to make them less aversive or scary.
Probably the 'ultimate reason' trapped priors happen has to be something about saving on computational costs, but IF it were adaptive/functional my guess would be that it's good not to update on certain phenomena – if you saw a tiger chilling out 100 times in a row and decided that tigers were mostly nonscary, you'd sort of be more correct than the tigerphobic villager but slightly less likely to survive.
But this seems wrong, because many things I get trapped priors about are totally harmless. Regarding BEC syndrome – weirdly, I've rarely had this for someone treating me badly. When someone is very mean to me it's "so extraordinary it needs conscious explaining", and I come up with best-fit explanations for their behavior. I do overall expect them to treat me badly again, but they feel like a complex system that requires complex thought in response. But when someone is MINORLY ANNOYING in the first few times I interact with them I can very trapped.
Thanks for breaking down a really important cognition "bug" so clearly. I want to suggest a small modification to your framing, which I think would add more clarity.
Your current theory treats agents as performing Bayesian inference on a stream of experience. If you instead assume the agents are performing active inference, the theory becomes more concise.
In your model, the agent receives a raw stream of experience that it must integrate with its priors. Active inference suggests that the agent queries the world, using its priors to generate hypothesis. The resulting experience is then integrated with the priors via a Bayesian update.
So a regular person locked in a room with a dog might have a low prior on the dog being dangerous. They are actively scanning the room for all sorts of cues including: is this dog dangerous, does this dog want pets, is this the cutest doggie they've ever seen.
A person with cynophobia is also actively scanning the room, but they are already certain that the dog is dangerous, so instead they are looking for exits and trying to figure out exactly when and how the dog will attack them. It's not that this person fails to do a proper update on the experience, it's that their experience is actually very different.
The queries that the agent performs are also not purely epistemic. They are not just trying to have more accurate priors, but to also act in accordance with their values. This captures both self-serving bias as well as emotions reducing the bandwidth of experience. In both cases, the agent isn't trying to learn as much as it can, but is instead trying to be safe, avoid pain, and maintain its social status.
I hope this made sense, and I want to credit Belobog with posting this first with his fisherman example.
I am also not sure that a perfect Bayesian learner would slide into confirmation bias, and my intuition says this is not the case. Unless every dog experience was *as terrible as you imagined*, your prior would gradually adjust down. Of course, humans are not perfect Bayesians. It's possible that humans would score "there was a dog and I was really scared" and "there was a dog and I was really scared and it attacked me" the same, or that they can't distinguish between a prior of 0.99 and 1, and so the update is always too small to move the needle. But I don't think we need to blame Bayesian updating when active inference is a much cleaner explanation.
As a spacecraft engineer who specializes in Kalman Filtering, which is basically recursive Bayesian estimation, I have always enjoyed the parallels with cognitive processing. This article reflects a problem we often see in KFs when the filter parameters are not properly designed, and the filter insufficiently weights new sensor inputs and simply propagates the state estimate using a math model (prior). This causes the estimate to diverge from the true state, or in other words, to deviate from reality. We say the Kalman Filter “has gone to sleep.” There are many possible causes for this divergence, and we don’t need to examine all the possibilities here, but the problem may be summarized as two types. The model used to propagate the prior state estimate forward in time may be too simplistic and needs to be improved, or there is some random element in the model that is not properly accounted for. These ideas, if applicable to cognitive processing, suggest that to avoid a trapped prior, we might consider expanding the dimensions or fidelity of our understanding, or else increase the uncertainty we apply to our prior. The latter solution is obvious and unlikely to be adopted since overconfidence is implicit in the problem of trapped priors. I wonder, however, if we might focus on improving our cognitive models, i.e., the mental model that produces our priors. Perhaps learning more about the dynamics of what we’re fearful of, for example “how do Rottweilers express themselves, and how are they stimulated to aggression or play?” Or, for reducing political demonization, “how do people of other political persuasions prioritize their value judgements?“ This is something psychologist Jonathan Haidt has emphasized and perhaps it is already part of standard psychological desensitization treatment. It seems possible to me at least, that improving our “dynamics model” of the thing we oppose or despise or fear may not meet with the same cognitive resistance as simple exposure to the results (measurement).
It looks like specific phobias (like cynophobia) are rare, for example this article https://www.med.upenn.edu/ctsa/phobias_symptoms.html says all specific phobias combined affect 7-9% of the population. I'm not sure what percentage is just cynophobia, but since most people have experience with dogs in their life and most people clearly do not get cynophobia, this means trapped priors about dogs are rare, most people can have a good and a bad experience with dogs, yet don't get trapped. If political trapped priors are exactly the same mechanism, I don't get why it seems like most everyone has them, instead of just small percentage of conspiracy theorists, etc.
There is a small flaw though on the chess piece illusion. I'm sure the actual illusion is based on the pieces being the same color but the brightness changing the perception. But, the pieces in the photo are actually darker vs lighter just as we perceive them. I pasted out a little section of both Kings lower part and the darker appearing one is actually quite a bit darker in reality, on the screen, compared to the lighter appearing one. Conceptually, though, the point is fine.
How about a stuck prior about a positive emotion? I'm not sure whether addictions qualify, but pursuing a hobby which isn't much fun any more because every time you think about it, you remember how much fun it used to be would count.
(off the cuff) I can't help but think that political partisans on the right, will interpret this post as "force feed conservatives LSD and party drugs until they become brain-addled progressives", with a side dish of "The Soviets were also a big fan of psychiatric treatments for having the wrong politics".
Great article, as always. I think, though, that you may be downplaying the role of self-interest (or at least self-aggrandizement) in this mode of thinking. You focused on politics, religion and phobias, areas where these thought loops seem to kind of “happen to” a person rather than being self-directed (OK, maybe not politics so much). The first thing I thought of, though, were tax protesters, and they don’t really fit the mold. I don’t know where I’m going with this, but that’s never stopped me before. Off we go.
For the uninitiated, tax protesters are a (very) loosely affiliated group of [redacted unhelpful insult] who believe that the federal income tax is unconstitutional and that they are therefore not required to pay it.
There are a thousand different flavors of tax protester, most of which philosophize about the difference between direct and indirect taxation, but the argument that brought them some mainstream credibility was first described in The Law That Never Was, a 1985 book by William Benson (I was hoping to do a book review on it for SSC, actually, but never got it together). The Law That Never Was claimed that the 16th Amendment wasn’t properly ratified in 1913, and that, as the 16th Amendment is what gives the federal government the power to tax US citizens’ income, no US citizen is legally obligated to pay federal income tax.
At its heart, the claim is that the various states that ratified the 16th Amendment ratified slightly different versions of it (so one state might have ratified “The Congress shall have power to lay and collect taxes on incomes, from whatever source derived, without apportionment among the several States, and without regard to any census or enumeration,” while another state didn’t capitalize “States” or DID capitalize some other random word, or used “remuneration” instead of “enumeration”, or a dozen other minor variations). Since three-quarters of the States are needed to ratify an amendment, and nowhere near three-quarters ratified any particular version of the above (both of which are true), the 16th Amendment lacks the force of law.
To attempt to tie this into Scott’s article: I think that if this sounds intriguing or plausible to you, it’s probably because you have a strong prior to having more money, and you’re ignoring the nagging doubts that otherwise would have made you laugh this off the way you would a Seventh Day Adventist. This is already going on too long, but: this argument fails because (1) most amendments were ratified with slightly different language since they didn’t have computers and shit to keep everyone on exactly the same page - if the Sixteenth Amendment isn’t in effect, neither is the First; (2) they considered this back in 1913 and the Secretary of State ratified the amendment anyway; (3) the federal government more or less already HAD the power to tax incomes before the 16th Amendment, so the Amendment doesn't mean what you think it means; and, above all else, (4) you aren’t going to walk into a federal court and convince a judge that the entire federal income tax structure is null and void and the government is required to go bankrupt, effective immediately, so cut me a check.
When I was in law school I got very interested in these sorts of weird legal arguments. Number (4) above is was intrigued me the most: how could anyone believe that this kind of legalistic thinking could have any practical effect? What was the end goal?
So I dug into it, I visited the forums where tax protesters hung out and swapped caselaw minutiae and jerk lawyers showed up to mock them. The community maps to Scott’s outline of “trapped priors” precisely. These were guys (they were almost ALL guys) who walked the walk. They weren’t hypocrites. They were in court making these arguments, despite being directly told by the judge that they were going to go to jail if they kept pushing it. And every time they were warned by the judge, every time another member of their community had their lives wrecked by brutal fines or prison terms, they believed in their cause more. The fact that nobody ever succeeded in avoiding the income tax by making these arguments seemed to reinforce the rightness of their cause rather than convince them that it was a losing proposition. I never really saw anyone change their minds, either through failure or through outsiders to the community explaining reality to them.
This was an awful lot to type when I don’t really know where I’m going with it. Sorry to anyone who read this far. I guess what I’m saying is: while my somewhat-extensive contact with tax protesters has convinced me completely that failing over and over again reinforced their beliefs rather than making them question their priors, the loop was also inextricably tied to self-interest, and would not have existed without it. They wanted money, and they wanted to be fighting a noble fight, in that order. They weren’t bitten by the IRS as a child and developed an irrational desire to fight back. I guess I’m asking: is it worth exploring the degree to which this thought process is actively selfish, and maybe approach treatment from that direction?
I think there's a better way of thinking about this. The political, scientific, and conspiracy examples of trapped priors have a better explanation, which I think is separate from the explanation for phobias.
Everyone has a model of where evidence comes from and how it gets into their heads. If this model is updated to say that evidence from source X is unreliable/biased/adversarially selected, then all subsequent evidence will be interpreted using that model.
If you come to believe that opinion X is supported by made up statistics and bad science, then you get stuck on the belief not-X. Your strong belief in not-X isn't what's causing the problem, the problem is caused by your beliefs about *how evidence is getting to you*. E.g. you believed GMO's are dangerous, and you also believe that any studies that disagree are funded by GMO companies and are therefore false. The way to change these beliefs is not to provide more studies that say GMO's are good. Instead you have to provide evidence that the studies aren't influenced by the interests of the companies.
In this explanation of stuck priors, your beliefs about "how the evidence reaches you" (meta-beliefs?) control how you update your object level beliefs. If humans were less computationally restricted, we would always be able to maintain many different meta-belief models, keep track of the probability of each, and for each meta-belief model we would have to keep track of the object level beliefs that your observations point to, *assuming that model is correct*.
I think the reason humans get stuck on beliefs is that it's *really hard* to keep track of multiple models without letting the object level beliefs formed under each model interact with each other and mess the whole thing up.
The solution to stuck priors, the way to get people unstuck, is to change their meta-beliefs. This is difficult and slow. For example, if a moon landing conspiracy theorist became friends with a few NASA employees, they would probably start to update their meta-beliefs, and get unstuck. But if the friendships were in any way deliberately arranged, then it's easy for the meta-beliefs to explain away the evidence as part of the conspiracy. The way to change these meta-beliefs is for evidence to sneak up from the side, from a source that the meta-belief doesn't automatically discount.
Under this model, there is no amount of evidence that would drown out the prior and cause an update, *if the evidence comes from a source which the meta-belief says is broken*.
This explanation doesn't help explain phobias and bitch eating cracker syndrome. I think they are a separate thing, explained by broken interactions between beliefs and emotions.
This is an interesting idea! However, it seems a lot (to me, as a math nerd) like trapped priors are an implementation error rather than a fundamental feature of Bayesianism. The fundamental reason why is that as more (say) "dogs are actually mostly friendly!" evidence accumulates, the odds ratio for "dogs are friendly" vs "dogs are dangerous" should keep going up, because Bayesian updating is basically multiplying the prior odds ratio by an odds ratio representing the weight of the evidence.
Related idea: the conservation of expected evidence (mentioned - introduced? - by Eliezer Yudkowsky at https://www.readthesequences.com/Conservation-Of-Expected-Evidence); what you expect to believe in the future is what you believe right now, if you're doing Bayescraft correctly. So if you were some person with a "dogs are dangerous" trapped prior, and you forecasted that future!you would believe that dogs are dangerous even more than present!you *no matter what happens*, then you should already have that level of belief in the dangerousness of dogs.
(The rest of the problems - like having beliefs about how dogs lead to unpleasant experiences, or how Democrats are morally evil - can be avoided by only holding beliefs about unambiguous object-level things that are outside your control, which avoids both ideological ambiguity and fixed-point problems/markets for entropy/etc.)
First, you've said in the past (e.g. in "Going Loopy" and "Can You Condition Yourself?") that the brain is in general pretty good at filtering out its own contributions when predicting things; it's not clear why this case is an exception (in the phobia case, I assume the answer is 'the thing that distinguishes *-phobia from normal fear is the failure to filter correctly', but if you're using stuck priors to also explain confirmation bias, then that's solidly in the domain of normal, non-pathological reasoning).
Second, why does the gradual version of exposure therapy work? In normal reasoning, a photo of a dog provides no evidence about whether I should be scared of dogs, a dog in a cage provides very little evidence about whether the dog would be dangerous outside of the cage, and so forth. Given that the cynophobe is supposed to be less responsive to evidence than normal, why doesn't that all get rounded down to zero difference from the prior, the same way they round away the much stronger evidence of being in a room with a Rottweiler and not getting bitten?
Third, the Drummond and Fischer paper finds that polarization increases on issues to do with political and religious identities, but not on issues that aren't politicized. That seems to me not like a trapped prior (if I can have a trapped prior about non-political issues like phobias, presumably I should also be able to have trapped priors about arbitrary scientific beliefs); on the other hand, it's perfectly consistent with a signalling/group identity type of explanation. Signalling also seems like a better fit for dog whistles; if I have a prior that Ted Cruz is malicious and he says 'I'm against New York values', then I might naturally update to 'New York values are good', or 'New York values are bad and Ted Cruz is lying about being against them', but 'New York values are a coded message for Jews, and Ted Cruz is conveying his anti-Semitism in a way intelligible only to his base (and also his opponents currently interpreting the dog whistle)' seems like a stretch. On the other hand, if I just want to make it really clear to everyone that I'm part of the Ted-Cruz-hating-tribe, then making an argument about secret codes does a much better job than providing a nuanced critique of one of his policies, because I might disagree with his policies for reasons other than partisanship, but I probably won't believe dubious theories about signals for non-partisan reasons. (Likewise, if I want to signal my partisan affiliation on the pandemic, I'm not going to look at a hard question like 'how much economic damage is worth it to mitigate spread'; I'm going to say something totally unhinged like 'Outdoor gatherings are a public health risk if you're having fun at the beach but totally safe if it's a BLM protest' or 'wearing a mask is about authoritarians conditioning us to submit to their orders' precisely because there's no non-partisan way to misupdate on evidence badly enough to reach those beliefs).
Somewhat related, in machine learning there is a notion of a “learning rate” which you can think of as the amount by which you update your prior based on new information. In the Bayesian outlook there is a precise right value for this parameter, but in practice people play around with it.
Interestingly, it’s often much better to underestimate it than to overestimate. If you underestimate the learning rate (which results in a tendency to stick to your priors) then you might not be as efficient with your computation or samples, but eventually you will reach the right point. In contrast, if you overestimate it, you may well never converge.
The above might be why it’s a decent heuristic to err on the side of under-correcting priors than over-correcting.
In the ML analog, trapped priors would be like a bad local optimum that we cannot escape from without making a large change. This also can happen sometime.
For many of these, it seems like there's something else going on--some kind of tossing out of outliers. If you have normally-distributed data and see some value that's +10 sigma out, it's almost guaranteed to be measurement error or something. But if your model is "normally distributed data," but the actual source of the data isn't really normal but instead some weird distribution that allows for +10 sigma events to happen pretty regularly, you'll filter out the problems with your model as being errors or lies or something.
When you tell me you saw a meteor shower last night, or a bad accident on the road today, my model says that's plausible, so I assume you're probably telling me the truth and may update my understanding of the world based on that. When you tell me you were abducted by aliens last night, or saw a giant moth flapping its wings over the city today, I'm probably just going to conclude that you're delusional.
When you make a plausible-sounding argument for medicare-for-all or shall-issue laws, I probably update my internal understanding of the world toward those being sensible policies. When you make a plausible-sounding argument for sacrificing babies to Moloch or re-imposing slavery, I probably update my internal understanding of whether you're a nutcase whom I should ignore instead of my understanding of whether maybe there's a good argument for baby-sacrifice or reopening the Atlantic slave trade.
In all these cases, what's happening is that your evidence/claim/argument is updating two different parts of my model, the "is the object-level thing they're claiming true?" and "are they a good source of evidence/claims/arguments?"
I don't think you really discussed it in the post, but an important characteristic of the types of scenarios that lead to trapped priors as you describe them is abstraction of goals. For example, the goal of a fear response to a dog is to avoid getting mauled by a dog. When you gather data from experiences with dogs, the thing your brain should* care about is whether the dog mauled you. But instead of updating P(I Get Mauled | I See A Dog), your brain updates P(Get Terrified | I See A Dog). So your brain is updating on an outcome that is, in a sense, one level abstracted from the thing it "should" care about.
This misalignment appears critical to to the phobia example, but I'm not sure if the same model makes as much sense in the politics examples. Using the same construct for some Policy X that my side supports, we'd have the brain updating not for P(Policy X Is Good | Evidence About Policy X), but instead for P(My Party's Policies Are Good | Evidence About Policy X).
I'm not sure how intuitive that sounds, which makes me wonder if the way the abstraction works in the two examples is different.
I'm not fully convinced that trapped priors can happen as a natural side effect of Bayesian reasoning. That is, I'm not convinced that purely "cognitive" trapped priors are possible.
As you note, when Bayesian sees evidence for X, they should never update in the direction of not-X. If you think that bad things are 90% likely to happen whenever you interact with a dog, and you have an interaction with a dog where nothing bad happens, you should update the 90% number to *something* lower.
Maybe only a tiny bit lower. But if it goes *up* afterwards, you simply aren't doing Bayesian reasoning anymore.
Perhaps you're doing some approximation to Bayesian reasoning that has this failure mode. But if so, it matters which approximation, and how it causes the problem -- if the approximation is doing all the work of explaining the pathology, then the treatment ought to focus on whatever is wrong with the approximation.
------
Related: in the dog phobia example, I think you may be conflating two things
- "how dangerous a typical dog interaction is likely to be"
- "how dangerous that particular dog interaction actually was"
The former is what the prior is about, and what gets updated to form the new prior. If you're Bayesian it should always *move* in the right direction.
The latter doesn't move, you just conclude whatever you conclude. It can *point* in the wrong direction, i.e. belief = dangerous while reality = not dangerous. But this one example doesn't become your entire prior for future experiences. Your prior is just whatever it was before, plus one update on a non-dangerous dog.
------
I want to connect this with jeff wade's comment about Kalman filtering. Kalman filtering really *is* an instance of Bayesian reasoning that can get off track and never really converge with the evidence.
What's the difference? Kalman filtering is about estimating a changing state, not a static fact like "how dangerous dogs are are a species." You predict the next state from a dynamic model, you make noisy observations, and you combine your prediction with the observation to estimate you actual state.
Here, the *state* gets updated, but the *dynamics* does not. If you predict "things get worse," and things look better instead, you will update to "things are currently [less bad than I expected, but worse that they look]." However, you don't update towards being less pessimistic about the future; you'll keep making pessimistic predictions and then partially correcting them, forever.
In the account of belief updating in this post, it sounds as if the prior gets updated twice -- first you combine prior and raw evidence to determine "what happened," then you combine "what happened" and your previous prior to get your next prior.
This is not how Bayesian inference works when you are estimatic a static fact. However, it's kind of close to what Kalman filtering does to estimate a changing state. The first step is normal Bayes to figure out where you are right now; the second step takes where you are now, and projects where you will be next.
I think that maps best onto human cases of continued engagement with one individual, like the example of abusive relationships. In such a case, someone could keep predicting "things will get worse" and getting corrected back to "things are the same," but never corrected towards more optimism. It might work for political groups if the belief is of the form "the Republicans are always getting worse" rather than "the Republicans are bad" It probably doesn't work for dogs.
(Hmm, maybe it's that every *individual* interaction involves some dynamics. Like, you always think the dog is going to get really mean in the next 60 seconds, and every 60 seconds of benign caninity makes you update to "it only got *slightly* meaner," and that still adds up over time.)
This seems like an especially good post to cross-post to LessWrong. I would love if you crossposted more in general, but this one is especially fitting.
Probably a dumb question: Why does the brain work that way?
Like, suppose you have sensations S1, S2, ..., S10 each of which is mild evidence against a trapped prior P1. According to this model your brain reasons from S1 + P1 to an unchanged (or stronger) prior P2, then from S2 + P2 to P3, and so on. And none of the sensations is enough evidence to overcome the prior, so things just get worse.
But in any reasonable framework, S1... S10 together are stronger evidence than S1 alone. And with enough sensations that should be enough to overcome P1. So the correct Bayesian math would be for the brain to update P1 + S1, S2, ..., S10 to a lower prior P'. In other words it combines its original prior with all the evidence since that time.
Why can't the brain do that? In algorithmic terms, is it a complexity / memory / "storage" limitation?
I guess one explanation might be that we don't actually remember sensation, just perception. (Partial because it begs the question "why not?") But that's hard to square with the political applications-- do we really "not remember" evidence regarding our political beliefs?
"...should try to identify a reliable test for prior strength [...] then observe whether some interventions can raise or lower it consistently. Its goal would be a relatively tractable way to induce a low-prior state with minimal risk of psychosis or permanent fixation of weird beliefs, and then to encourage people to enter that state before reasoning in domains where they are likely to be heavily biased."
Isn't it currently a Dark Epistemic Art? That is, to see the best available method, shouldn't we just check how people that succeeded in convincing others of really weird shit do it, and then try to replicate the results but without pre-determining the conclusion?
I think that the ideas you've presented relating depression to placing low priors on sensory evidence might explain the apparent high rate of trapped priors in politics.
We have a word for people who place a low prior on sensory evidence: rationalists. The word is Latin, but the division of the Western world into people who trust sensory evidence (empiricists) and people who don't (rationalists) goes back to Thales and Parmenides.
Rationalists always conceal the existence of this division when they dominate public discourse, and we're in one of those rationalist-dominated periods today, despite the continuing successes of empirical science. So today, most people think "reason" and "rationality" are synonyms, when in fact "rationalism" is an oversimplified caricature of reason which states that:
- the world is split up neatly into categories by the words of our language
- every statement expressed in those words is either True, all the time, everywhere; or else it is False, all the time, everywhere
- Reason means beginning with a set of axioms or divine revelations, then applying logical deduction in order to deduce new Truths
- you can believe anything you've deduced with 100% certainty
Empiricists approach politics as a practical matter, in which we have a limited amount of resources to allocate in a way that will optimize some measure of utility. Rationalists approach politics as a moral matter, in the mindset of Plato when writing Republic.
It so happens that, today, every political party is rationalist. (This is usually the case. Empiricists are usually out discovering stuff, making stuff, or trading stuff, rather than playing politics. Democratic Athens, the Republic of Venice, the Netherlands and some Italian city-states during the Renaissance, and the American Revolution are the only empiricist states / political movements I can think of at the moment.)
- All Christians are rationalist, because Orthodox theology, and to a lesser extent the New Testament, were based on Plato.
- All Hegelians are rationalist, because Hegel was very Platonist. (Even if you disagree, you still gotta admit he was a prototypical rationalist.) This includes Nazis, Marxists, and Progressives. "Progressive" doesn't mean "a person who wants to make things better"; it means someone who believes in the divine march of Progress, under the guidance of the World Spirit, as explained by Hegel, to achieve the ultimate "perfection" (a Platonic concept) of the world.
- Libertarians are rationalist, because they have one supreme value (liberty), which takes priority over all other values, at all times and in all places.
- For the same reason, environmentalists, feminists, post-colonialists, and almost any "one great cause" movement is rationalist.
Rationalists are by definition people who place low value on empirical evidence. This is why Christians still won't believe in evolution, Marxists don't care how many times Marxism fails disastrously, and the "defund the police" movement doesn't care that excluding the police from the CHOP in Seattle for 3 weeks made it the most-violence-prone place in all of America for those 3 weeks. Theory has epistemological priority over observational evidence to rationalists.
Mechanistically, trapped priors may be related to the phenomen when memories become experience. So, one spend time with a dog, the conclusion that it's terrible is confirmed. After that one recalls the experience, and brain uses the posterior+real experience to generate new "memory" which replaces experience. This biased memory is then combined with previous priors to give new and worse posteriors (which becomes prior next time one recalls the events). And so every time one recalls something terrible, it becomes even more terrible. And if the event is "triggering", the recall/reconstruct cycles are accelerated, one thinks about terrifying events much more frequently, and the process of replacing real memory with biased ones is very fast.
The description of trapped priors sounds a lot like Lewis's reply to Hume on miracles: having decided a priori that miracles are impossible, Hume dismisses the (rather large) volume of testimony about miracles. He cites the credulity of the witnesses as evidence against their reliability (why credulous? c'mon, they're talking about angels and demons and stuff!), and then concludes from the absence of evidence that miracles are impossible.
Make of Hume's argument what you will, but it's a striking structural similarity to the trapped prior phenomenon.
I studied neuroscience in undergrad. I recall the brain having fairly idiosyncratic machinery for... well, everything. Including fear and anxiety.
Like if you were going to *design* a brain with an anxiety-processing center called an amygdala, you'd probably figure out what inputs the amgydala needs and just... give it those inputs. But no, the brain takes perceptual information and sends it to the amygdala via two different pathways: a fast path that goes straight to the amygdala, and a slower path that goes first to the cortex and then to the amygdala. (https://en.wikipedia.org/wiki/Joseph_E._LeDoux#Work_on_threat_response,_anxiety,_and_emotions)
Does this have implications for fear and trauma and memory? LeDoux thought so! (Though my information is dated now.)
My point is, I don't think the brain is "just doing bayesian updates." I don't think that all neural processing is to predict things and then update those predictions in a bayes' theorem-esque manner. (I know Scott has written about this in the past. I'm not sure to what extent he thinks this is literally true.) I think that the brain is messy, and that no matter how the low-level computation works there is some really weird and idiosyncratic "programs" that are built on the low-level stuff, and at least some of that is involved with fear and looks very little like Bayes' theorem.
I think this post goes too far in the "everything is Bayes" direction. It's an attempt to unite disparate observations, which is great, when it works. I don't think it works here. It seems too pat. And not very predictive.
----------------------
Those are my overall thoughts. I also have a grab-bag of random things that came up while I was reading:
1. This post seems to imply that a lot of biases can be described as "bayesian reasoning + up-weighted priors." You've got: bias in favor of one's tribe, self-serving bias, and I imagine that "bias in favor of thinking good things about oneself" would also be included.
That said, what is this post meant to explain? If the theory doesn't logically entail that humans will have *particular* biases, then it's not very helpful. Why do people have "trapped priors" when it comes to their tribe? Why not some other thing?
Also, if "trapped priors are purely epistemic and don't require emotion," why are all of our good examples of trapped priors related to emotion? Maybe there's more going on here that this simple theory can't explain.
2. If the theory is meant to suggest ways to "unstick" sticky beliefs (say, by using psychedelics), do we expect psychedelics to affect all of the biases that (we claim) result from trapped priors? Will people show less self-serving bias? This strikes me as predicting too much.
3. A number of people in the comments are now saying "I have a trapped prior for thing X," meaning, "I have trouble changing my thoughts/feelings about thing X."
But this is post hoc. The trapped prior isn't predicting anything. "Trapped prior" is becoming a name for a thing, but not a *cause* for a thing.
This is, obviously, not Scott's fault. But the point should be made that we need independent evidence for calling something a trapped prior, and the fact that we don't have that is currently a weakness of the theory.
4. "...you only habituate when an experience with a dog ends up being safe and okay. But being in the room with the Rottweiler is terrifying. It's not a safe okay experience."
By this logic, it would be very difficult for the rats in your example to habituate to the bell, wouldn't it? The bell would cause them to predict pain, just as the dog causes some people to predict pain.
(Yep, I studied neuroscience in undergrad and an explanation like Scott's made sense to me at the time. Now, I don't really think it does.)
5. "...[psychedelics] can loosen a trapped prior, causing it to become untrapped..."
What does "loosen" mean in statistical terms? Push a prior toward uncertainty (.5)?
This question is less because I want an answer, and more because if everything comes down to Bayes theorem, then it would be helpful to *use simple terminology that applies to Bayes theorem* when talking about this theory. "Loosen" is very vague.
Not really the point of the post, but I think it’s very plausible to think that joe Biden personally doesn’t support rioters but most people in his administration — starting with Kamala Harris — do. How else would you interpret tweeting out a bail fund for rioters?
I think that vanishingly few political disagreement are actually about facts, so it's hardly surprising that evidence about facts is unlikely to change them.
Facts are tiny side-issues that people spend a lot of time arguing about, but which are usually pretty tangential to the point. If Alice and Bob sit down to argue about the death penalty, then Bob can bring a whole bunch of studies that show that the death penalty reduces crime rates, and Alice can bring a whole bunch of studies that show that the death penalty actually has no measurable effect on crime rates, and they can both spend many hours arguing about the relative merits and flaws of these piles of studies.
In the end, though, this debate is all for nothing -- Alice disagrees with the death penalty on principle, and even if you convinced her that the death penalty _does_ slightly reduce crime rates then she's not going to change her mind, she'll say that a small increase in crime rates is a small price to pay for the abolition of that horrible barbaric punishment. (Nonetheless, she will fight tooth and nail against having to admit that it actually _does_ increase crime rates.) And same deal for Bob.
Or, if you listen to libertarians, you will hear a great host of arguments about how getting the government out of such-and-such would lead to improved outcomes. I have never once heard a libertarian say "Look, to be honest, government interference in XYZ actually probably has positive outcomes, but I still think that the government should get out of XYZ on principle".
Are increased minimum wages good or bad, overall? It doesn't matter, because everyone seems to either support them on principle or oppose them on principle.
Why do people insist on debating facts when their disagreement is about values? Possibly because they are arguing for the benefit of some hypothetical listener who might have no particularly strong value opinion on the matter but are pragmatically inclined towards good outcomes. And maybe these people do exist out there in the middle somewhere, but they tend to be relatively quiet. Among the sorts of people who debate politics on the internet, though, meaningful fact-based disagreements are practically absent.
BECS is probably the thing that generates the most ill will. As I'm on the right, I constantly hear that so-and-so - or I - is racistsexisthomophobe because of innocuous comment X. I don't think there is anything so guaranteed to cause ill will as "Oh, you _really_ mean X when you say Y."
(That said, I've tussled enough with White Nationalist types to be aware of Scott's point on Weak Men.)
The best thing I can think of is that, even if you think this, not to say it. Try and engage with the literal meaning of the words as best as possible (the late Hitch was good at this). And if you must say "This looks like X to me", explain that it is something you suppose, and why you think it, not just say "You really mean X".
For example, I've dealt with a lot of people who have nothing good to say about Israel, and people who have nothing good to say about the new South Africa. And who in both cases offer euphemisms about the terrorist movements each face (yes, S.A. has its white nationalist terrorists; they are called the AWB). Now I cannot say that each of these people is anti-semitic/racist, but what I do say, if I hear enough of this, is to say "Look, this is giving me a really bad feeling, and here's why."
So there are multiple ways to have trapped priors? Let's take a group of people that have a strong trapped prior that dogs are dangerous:
* Alice suffers from a weird sensory disorder that makes dogs look really dangerous.
* Bob suffers from a weird brain disorder that makes him unable to update his prior with new information.
* Carol has cynophobia. A dog "is so scary or hated that it's aversive to have to perceive it directly, [Carol's] mind decreases bandwidth on the raw experience channel relative to the prior channel so that [she] avoid the negative stimulus."
* David hates dogs for political reasons. Once again, a dog "is so scary or hated that it's aversive to have to perceive it directly, [David's] mind decreases bandwidth on the raw experience channel relative to the prior channel so that [he] avoid the negative stimulus."
Can we think of more cases? How much does these four cases really have in common? What is the actual difference between someone who has a strong prior that dogs are dangerous and someone who has an equally strong but *trapped* prior that dogs are dangerous?
Perhaps we need an easier approach. For example, I have found it helpful to ask students, "Are you willing to accept that your idea is wrong if you are shown evidence that it is wrong?" Prime the person to overtly and rationally accept the possibility of change. Have you or other workers in this area tried that?
Also, you are looking at behavior change, but there is a pretty solid body of evidence that behavior change is more enduring when the new behavior occurs at higher frequency with less latency. You and others ignore frequency, as far as I can tell. If you want people to change responses, it might be more effective to train them to change a small response quickly and then do it repeatedly. Stop looking only at quality and look at frequency.
I will give you an example. If you want your client to stop feeling intense fear of dogs, do something to help her quickly change the feeling. For example, give the person a picture that causes a fear reaction, then teach her to change that feeling quickly. I don't know your clients, so I don't know what would do that, but you could possibly give a mild fear stimulus and then very quickly show a picture of a dog or some non-doggy but delightful, relaxing, or humorous stimulus that would quickly alleviate the feeling of fear. Alternatively, you could teach a mental reaction that would relieve the fear and teach her to quickly engage this reaction (e.g., a statement of some kind). Then have her do this more and more quickly for a minute or so, using multiple stimuli and trained reactions or competing stimuli.
This is just an outline of an approach that might work. The main point here is that frequency and latency are both important, and as far as I can tell, you are not much aware of that.
I agree with most of your analysis, but I've met a whole bunch of people for whom "punish women for being sluts" was pretty much the explicit reason of choosing a pro-life position. Typical mind fallacy.
From the mathematical point of view it seems to me that there are two intertwined but different issues.
1) You start with a prior that is extremely narrow, a Dirac delta for any practical purpose, or maybe just something with a finite support (i.e. zero outside a certain interval). This means that even if the Bayesian updating works perfectly you can't escape the (for all practical purposes infinite) prior information that you start with. By the way "prior with finite support" is what came to my mind when I read the phrase "trapped priors".
2) The updating algorithm deviates from perfectly Bayesian and can actually update the prior in the wrong direction (it could also be Bayesian at the core but with some inbuilt "data preprocessing" that amplifies evidence in one direction disproportionately).
Maybe a similar point has been raised in other comments (I haven't read them all)... Anyway it's a probably superfluous "technical" comment that doesn't change the take-home message of this brilliant analysis.
Hah, we just covered this in a course I'm taking. Our model was like this: a queue of people have a prior that restaurant A is better than B with 51% probability. Now each person on the front receives a signal that one of the restaurants is better than the other with 52% probability.
If the first person receives the signal A>B, then everyone in the queue will end up going to A: When they're at the front, they have their prior, their signal and they can infer the first person's signal is A>B (since they ended up going to A). Combining this information will always yield A>B regardless of their own signal.
"Trapped prior" is a good way to describe Scott Alexander's attitude towards the New York Times. The sensory evidence (the NYT being full of people wanting to destroy Scott Alexander's life and not afraid to lie in service of that goal) has been swamped by the prior (the NYT is the font of all that is true and holy)
I feel very skeptical about entirely-internal-belief-feedback-loops but more open about a mixed internal+external-belief feedback loops (or local minima in belief-space which, say psychedelics might help with)?
Say, if you are a partisan Democrat and have a discussion with a partisan Republican probably the interaction will be very bitter and shouty and you will develop more of an ugh-reaction to Republicanism
Consider the hypothesis "you are in a russian roulette environment." Ie an environment that has some small chance of killing you.
So immagine your priors say, either dogs are safe and quiet, or dogs bark a lot and occasionally kill. Then when you see a barky slobbery dog, your belief in their danger goes up. Maybe the hypothesis "dogs are barky but safe" is in there too. If the chance of dogs killing you in the dangerous dogs hypothesis is 1%, thats going to take many 100s of very scary dog visits before it shrinks away. (Enough that you would expect to be dead if dogs were that dangerous)
So with these fairly sane priors, and perfect baysian updating, you get the behaviour where they walk away from a dog uninjured, but are more scared of them.
Of course, many things look baysian with strange enough choice of priors.
The doomsday cultists. Ok, they are just not baysian, but that could be a lot of selection there, in that the least rational are most likely to be doomsday cultists.
Don’t be so down on cognitive trapping; it can have big benefits too! It’s fast (all precomputed, no serious thinking needed in the hot path in response to stimuli) and it’s immune to lyin’ liars (because no matter how eloquent they are you’ve presupposed their falsehood).
And there’s all sorts of reasons you might want fast and durable mental tools. You want them if a tiger is hoping to eat your face. You want them if an abuser is trying to gaslight you. You want them if you’re being tempted away from the high road. All things where you don’t have time to think and it costs a lot to be wrong.
It’s sort of the opposite of the scientific method, which is all about focusing on exactly what stimuli should flip your prior. But science sometimes takes decades, or lifetimes, or more. Time enough to cut through the noise.
The other way through cognitive trapping, I believe, is humor. My father taught me that the way to defuse any situation is with a gentle joke. Perhaps if you’re laughing with everyone then you can’t register the experience as negative? And therefore you can begin to update your prior in a more Bayesian fashion? I do get the sense that my father’s use of humor has more to do with him being a middle child and less to do his neuro PhD or his pediatric neuro MD, but I’ve come to believe in his method.
Aside: one of the really fascinating things about working with geniuses - not just very smart people but geniuses - is that they develop a strong cognitive trap that most people are wrong and they are right. It’s not totally irrational! Except when it is.
An interesting analogy to this issue might be "regularization" in machine learning (neural network) frameworks. To prevent over-fitting (i.e. a network that learns irrelevant patterns or "noise" in its training set), data scientists optimize a fitness function that includes some features of the network itself.
One common regularization option is L1-norm regularization, which penalizes the network's fitness score by the sum of the absolute values of its weights. This normalization encourages networks to push as many weights as possible towards zero, forming a sparser (fewer-factored) explanation for any inference.
Shifting back to "trapped priors" and using this as a metaphor for human reasoning, habituation requires us to replace a simple explanation ("all dogs are scary") with what is at first a more complicated one ("almost all dogs are scary, but this dog isn't scary because it's licking its own rear end right now.") If we suppose we're instead applying a strong preference for a simple rule, then a simpler explanation that's less accurate ("all dogs are scary, and also this dog is scary") may still be preferred. Habituation would then work because the strong belief is not directly challenged (even if all dogs are scary a *picture* of a dog can't be scary), so the preference for a simple belief is not triggered in the same way.
> I can't unreservedly recommend this as a pro-rationality intervention, because it also seems to create permanent weird false beliefs for some reason, but I think it's a foundation that someone might be able to build upon.
It creates weird false beliefs because lowering the prior doesn't remove the error.
There's a type-I/type-II error tradeoff here. Stick more strongly to (typical) priors, and you'll be less of a conspiracy theorist but more vulnerable to failing to notice that the building pandemic really is that bad. Stick weakly to priors, and you'll be quick to panic about COVID in January 2020 but also become convinced that lavender cures autism or whatever. The larger the errors in the inputs, the more important this tradeoff becomes.
Since my whole mission here is to figure out grey-tribe failure modes, here's a speculative one: the grey tribe likes to think about weird ideas because it thinks bad ones will fail and good ones will succeed. Smart grey tribers have relatively low error in their analysis, but the error is heterogeneous, and if you consider enough weird ideas you will eventually land on one where your error convinces you of a false premise strongly enough to lock. And the tradeoff can't save you: low-prior people are more vulnerable to the error, and high-prior people are more easily locked into it.
You've written about autism/schizophrenia being at each end of the evidence/priors spectrum. Since schizophrenia is "the one with the hallucinations" (sorry for this terrible oversimplification) I find it weird that psychedelics, "the thing that causes you to have hallucinations" (again, sorry), map onto the 'evidence' end of the spectrum.
I think the problem is that there are two fundamentally different things that superficially look like giving more weight to evidence:
1. One is actually upregulating the evidence channel vs the priors channel. EMDR and mindfulness may work this way. Seeing pulsating objects and auras while tripping also seems to go here. Because your priors are downplayed, low-level quirks of your visual system come through unaltered. Evidence is given more weight, so colors appear more vivid and saturated.
2. The other phenomenon is akin to increasing temperature in the annealing metaphor. Because higher temperature makes the prior landscape effectively flatter, you need less evidence to get out of local optima, but it also means you jump around a bit. So, even if you don't particularly give more weight than usual to your senses, it may be easier for a cloud to induce a fleeting experience of pareidolia - and it's gonna be equally easy for you to snap out of it and fall into a different, random attractor state, and then another, and so on until the LSD wears off and your prior landscape looks rugged again.
To sum up: in this model, autism and schizophrenia would be variation along the first dimension and independent of temperature; psychedelics would be affecting both knobs, making you pay more attention to evidence _and_ making your priors more malleable.
Not sure any of this makes sense, but at least I now have a better understanding of my own confusion :)
I believe the framing a person applies to an issue is capable of overriding everything else, and that this is the most common way trying to be an educated, informed person goes wrong. This is not only a trap; this is THE trap, as long as you are human. I'd say most important sources of stupidity that originate in otherwise brilliant people are just manifestations of them framing or contextualizing information in such a self-serving, or, as you formalize it, in a prior-confirming way. I think almost everyone has a natural tendency to do this, especially with emotionally charged issues. (There are simpler mistakes, but those are fixable)
I think a primary exercise of having a healthy perception of the world should include ensuring that one remains humble and attempts to question their most entrenched worldviews as an active practice. This probably includes a mixed information diet from sources that have many different perspectives.
Maybe constantly trying to undermine one's own worldview from all sides is actually the most essential practice if you want to have the best chance of being correct about things. This could be the real central praxis of rationality, even more so than dealing in certain mathematical formalisms (which, while great, can easily be used in service to one's ideology)
Supporting the idea of trapped priors independent of any emotional component, you can very easily end up with a trapped prior in a naive computer implementation of a Bayesian filter that uses normal likelihoods instead of log-likelihoods: if your prior becomes small enough, the floating point value will underflow and turn into zero.
In theory, it can happen with log-likelihoods as well, as IEEE floating points can become infinity if sufficiently large values are added together. However, the amount of evidence you need to gather for this to occur is quite obscene.
Have you considered a trapped prior might actually be an extremely useful tool to have?
Suppose the trapped prior is 'everything will be ok'. Whenever a situation seems to be not ok, if that prior really is trapped, you'll rely on it, which will make you feel that everything is, indeed, OK, and as a result, you get more evidence for the trapped prior.
This seems like a pretty decent description of what religious faith gives people. If you have 100% confidence that things will be OK, that they happen for a reason, and your job is to always accept your circumstances and do the best you can, then you can likely interpret any situation - even ones most people would say are very awful - as being OK, so long as that prior is strong eonugh.
The only hing I don't use this prior for is to anticipate that there will be no bad consequences from a possible course of action. It would obviously not be good to say, "i won't buckle my kids' car seats because nothing bad will happen." I don't' think most religious people are doing that, but i can see how this sort of thinking would not be great at convincing people they should wear a mask or want to reduce carbon emissions, for example.
But if you're crippled with anxiety and fear about the world, a trapped prior that says "i will be ok" seems like an extremely useful tool.
"I don’t know why this doesn’t happen in real life, beyond a general sense that whatever weighting function we use isn’t perfectly Bayesian and doesn’t fit in the class I would call “reasonable”."
Assuming the model, it seems to me that the reason for the problem is that there's a cycle. In your diagram, the gray box isn't where the Bayes happens. It's just some sort of weighted average function between experience and priors. The Bayes happens between the perceptions and the priors. But since perceptions are modulated by priors, this is factoring your priors into your evidence, which is a little silly. After all, it means you can update on nothing: take a 0 experience, average it (in any way) with your prior, and you'll have a non-0 perception to update on.
Most of the time, this probably is fine, because it just makes you update slightly less or more than you should, and in some cases could be helpful (for example, if, say, your brain isn't capable of making large enough jumps in prior in a single bound, then having your prior reinforce itself lets that happen over time, maybe). In some cases, it'll cause problems, like if the average between a large negative prior and a small positive experience becomes a small negative experience, the opposite of the actual experience. So, a proper Bayesian would have to do the updates on prior and experience *before* funneling that into perceptions. Or maybe do updating and perceiving completely separate. At least, the perceptions part would not be an input into the Bayes function.
WRT the "cynophobia" thing, what about cases where dogs truly do behave abnormally aggressive towards a person?
I've long since lost count of the number of times some person, innocuously walking their innocuous dog near me, has ended up totally shocked as the beast goes into a rage, straining at the leash trying to reach me and attack me. They always apologize profusely, saying the dog *never* behaves that way. Well... for me they do, pretty darn consistently.
In a similar vein, I have never in my life met a mean cat. Not even ones whose owners warn me about how mean the cat is. I can always get them to cuddle up in my lap and purr as I pet them with no trouble. Again, this frequently surprises the owner; watching the responses of others is a good way of getting outside my own biases and knowing something real is going on.
Maybe I just smell like a cat on some pheromonal level or something?
I guess my question would still have to be why this actually happens. Is there just too much friction in the system? Is the brain using some sort of computationally efficient approximation of Bayes Theorem that can get caught up on things like this?
Conspiracy theories are another case of this, where they can fully explain the lack of evidence ("because it's a conspiracy!") and become trapped.
More broadly, priors are not just a number -- they're a number attached to a whole mechanistic theory about the world.
If I tell you an urn contains a red and white ball, and then I pull a white ball out of it, then when you see the white ball, your confidence that the next ball will be red goes up.
If I tell you I filled the urn with balls of a single colour, then when I pull a white ball out, your confidence that the next ball will be white goes up.
The same observation, but you update in different directions depending on your prior!
The issue with trapped priors might not be that the updating process is irrational, but that the prior itself was way overconfident to begin with. A properly ignorant prior will contain some seed of doubt in any conspiracy theory, which grows faster on any evidence of no conspiracy than the conspiracy theory's cover-up likelihood. But if you don't have that seed of doubt to begin with, a prior of zero still updates to zero no matter how much evidence is in its favour.
While we're at it, consider how you update on the evidence in these cases.
People say economists are always predicting stock market crashes that don't happen. (Until they do). So some people start to treat the history of past wrong predictions as evidence that decreases their confidence in any further predictions. And I can't say they're wrong to do so.
But what about earthquakes? After a seismologist starts to worry about a build-up of stress in a fault line, and sound a warning about a potential earthquake, each passing year in which that prediction rings false might *increase* the seismologist's confidence that the earthquake will happen the following year. And I can't day they're wrong to do so either.
This is because their model is of a stress value climbing and climbing, which triggers at a threshold that is known to very low condidence. When you start to think that stress is within your trigger range, you start to worry about an earthquake, and when it doesn't trigger you worry even more.
But that's also, more or less, the model that economists use to predict crashes.
You don't need psychedelics to get permanently trapped weird beliefs. Claiming that they are false, however, seems excessive. I have a friend who believes in big foot. He seems to be oblivious to the lack of evidence. But then *I* don't believe in big foot, so my evaluation of the evidence is different from his.
FWIW, I tend to think of psychedelics as something that relaxes the error checking protocols of the brain. If these "trapped priors" are a manifestation of the error checking gone awry, then the whole thing makes sense. But note that the error checking exists for a valid reason. That the mechanism isn't perfect doesn't mean it isn't important.
ISTM that your line of reasoning would indicate that the best way to convince someone with a trapped prior is to present them with lots of REALLY weak (and unemotional) arguments that they are correct. I've heard this approach suggested before, but it's really difficult to intentionally create weak unemotional arguments that you believe are incorrect.
One thing I'm noticing here is that all the priors you mention seem to be beliefs about the believer's own identity. A Democrat can't believe Republicans are right because they believe that they (a Democrat) are right. The cynophobe, at some point, isn't afraid of dogs any more, they're afraid of themselves--specifically, their own reaction to dogs. The cultists ARE reacting to evidence when prophecy fails and they double: they're reacting to the evidence that faith has been rewarded with acceptance within the cult, and a strong counterexample to the cult's belief system is an opportunity to demonstrate greater faith, thereby gaining greater acceptance. There's a reason faith being rewarded with acceptance is part of the memetic code of many religions: it's an effective "immune system" which defends the religion against outside evidence.
The coyote/polar bear example is a harder stretch, but as someone who was rewarded repeatedly growing up for "being smart", I definitely have some beliefs about myself which cause me to react negatively if ANY of my priors are challenged, simply because I've been conditioned that acceptance comes from having correct priors. I think this is a fairly universal cause of irrationality, but those of us who have been rewarded for being right a lot perhaps are the worst afflicted.
The interventions you mention fit this pattern to some extent: putting the cynophobe in the room with the Rottweiler immediately addresses the fear of the dog but does nothing to address the fear of the cynophobe's reaction to the dog--the slower exposure addresses the cynophobe's fear of their own reaction. Psychedelics are correlated with decreased sense of self, even to the point of complete ego death. In some Eastern practices, decreasing sense of self is an explicit goal of meditation (though I see this de-emphasized in adaptations in the individualistic West). Christian meditative practices do tend to have some giving up of the self, though usually the goal is to supplant yourself with god--but I can see this opening people up to beliefs that don't oppose their religion (i.e. the coyotes/polar bears example). Diet, exercise, etc. might help because they shift the balance of fight/flight hormones: it seems apparent that the same hormones that regulate fear of harm to the physical self would regulate fear of harm to the psychological self.
Showing my hand here: there's a number of points here where I'm well outside my own areas of confidence: i.e. on the hormone idea I'm maybe at 20% confidence. I think I can develop a fairly confident position based on a bunch of weak pieces of evidence (i.e. 10 studies with P=0.7 results actually result in a P=0.03 confidence) but it would be easy to change my mind with a single stronger piece of evidence.
"But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it."
This part worries me a little, specifically that first sentence and the notion that there is some fundamental underlying reality that we can access and know. Inherently the bulk of political disagreement tends to revolve around things that the "scientifically correct"* answer is "we don't know", as opposed to things like "There is a polar bear visible from an LA freeway." The bear thing is a lot easier to develop evidence towards proving than say whether or not gay marriage is a better policy than civil union laws or something. The latter probably requires some robust experimentation over a long period of time to judge properly, where as the former has evidence about that is reasonably easy to get.
Yet people often treat these questions as binary True/False statements instead of Maybe? My suspicion is that this is because for many things an answer of Maybe implies a course of action similar to False: do nothing right now. This stands in direct opposition to those who believe the answer is True, or at least want people to do the things True implies should be done. So you are a badwrong climate denier if you are skeptical.
On the other hand, Maybe looks dangerously close to True from the standpoint of some who believe False. You might be willing to do a little bit of something that shouldn't be done, after all. Give an inch, and those bastards on the other side will take a mile. So you are a badwrong libertarian anarchist if you think there might be some virtue in taking a look at police accountability.
To me, this suggests a problem with politics in general: no one pays the price for their beliefs, so extreme beliefs are optimal. We may dress them up as scientific or evidence based, but most evidence actually points to "We don't know" or "Don't do anything, ever", so it becomes a big game of rationalization, not rational decision making. An external, testable reality has just about nothing to do with it.
I wonder if the processes you describe are related to the phenomenon whereby a trauma is made worse by how the people around them view the traumatic event. For instance, I recall reading that rape victims who are surrounded by people who consider rape as the worst possible thing that can happen to someone, generally fair much worse than those who aren't in such an environment, ie. people treat their trauma like other terrible circumstances that befall people every day.
Theories I've read suggest that in the former case, the trauma becomes a more central part of their identity, and so they cling to it in some way, but I never quite understood the mechanism behind how this works.
While you were describing how the brain attenuates the sensory input of the traumatic event, and how this just reinforces your trapped prior, this seemed kinda-sorta similar to the trauma scenario above: maybe you don't remember all of the details of your trauma with perfcet clarity, but you have a context (the people around you) constant reinforcing the view that your trauma was the worst possible thing that could happen to a person, and so that heightens your fear of it and traps you in this prior that you are a victim of horrendous abuse that will forever haunt you.
Couple that with the facts that memories are at least partly reconstructed (presumably priors factor into this as well), and I can see how the environment in which a victim is immersed could quite readily affect their recovery and how they cognitively engage with the world.
I nominate a fourth type of bias as equally significant to the three Scott highlights (cognitive, emotional, self-interested). Social bias warrants its own category, I think, because it is so much more than mere external context, -- since society plays such a reifying role in our perception of ourselves, and vice versa, making it a literal extension of ourselves -- and its gravitational power is simply enormous. No doubt one could use the trapped priors model to analyze societies (and subsets) as well as individuals.
I plead guilty of course to all of the above -- but I'm not 100% certain it's pathological. I admit I have a bad bitchy crackers problem, say, watching films of Hitler patting a dog (in fact, every fucking thing he does STILL annoys me!). And I would think we'd have to gear up into some new quantum-leap kind of stage of human development before we'll be collectively ready for a fully rationalist approach where we'll effectively view someone like Hitler without emotional bias during the dog-patting, and simultaneously recognize that he indeed was one of those rare polar bears that shouldn't be denied or ignored. As emotionally painful as it was, it seems relatively healthy that society (at least western, writ large) developed a severe enough trapped prior, lasting more than 50 years, that our "irrational" fear of brutal populist authoritarians actually helped keep them at bay for that amount of time. (Which brings up the question of time passage, at least as a factor when it comes to releasing societal trapped priors.)
"but God coming down from the heavens and telling you anything probably makes apocalypse cultism more probable, not less.)"
There is actually a Talmudic story where God speaks from the heavens telling the majority party of the Rabbi that they are wrong, and their response is "Butt out." (It is not in Heaven. You have told us to follow the majority, so that's what we are doing.)
I believe the source for the evidence that more scientifically sophisticated people are more likely to agree with their side is Dan Kahane, although someone else may have produced similar results. His view is that it is rational behavior. Whether you believe in evolution or catastrophic anthropogenic global warming has almost no effect on the world, since you are only one person. But it can have a large effect on how well you get along with the people around you who matter to you. So it is rational to talk yourself into the belief those people hold even if it is wrong. The more sophisticated you are, the better you are at doing so. This is a little like your rich man/taxes example, but not quite.
For some reason, I can no longer reply to anything except for the top level where I am starting a new thread. I assume this is some sort of bug and may be affecting others.
To me, this sounds very much like a more rigorous exploration of Kolakowski's Law of Infinite Cornucopia "…. for any given doctrine one wants to believe, there is never a shortage of arguments by which one can support it.
A historian’s application of this law might be that a plausible cause can be found for any given historical development. A biblical theologian’s application of this law might be that for any doctrine one wants to believe, there is never a shortage of biblical evidence to support it. "
>>"Along with the cognitive and emotional sources of bias, there's a third source: self-serving bias. People are more likely to believe ideas that would benefit them if true; for example, rich people are more likely to believe low taxes on the rich would help the economy.... I don't consider the idea of bias as trapped priors to account for this third type of bias at all; it might relate in some way that I don't understand, or it may happen through a totally different process."
Maybe I'm misunderstanding, but I'd see "self-serving bias" as following from the model in an almost formal sense, once we add the stipulation that the experience you're integrating with your priors is always *your* experience.
Insofar as, e.g., low taxes are in your (narrowly construed) self-interest, then every time there's a tax cut your direct experience will be of a positive outcome. Maybe you also see some studies or hear some anecdotes about the tax cut being bad for other people, but these seem dubious. Your prior is that tax cuts help people, so if these other people aren't being helped, isn't that more likely to be for some exogenous unrelated reason?
Of course, if you encounter enough evidence of the latter kind, you might eventually come around to the view that your case is exceptional and the optimal policy *for you* isn't the one that's best for society on net.
The point is just that your own immediate experience of [Policy X --> Positive Outcome] is always going to introduce a certain initial weight drawing you away from reflective equilibrium. Which is precisely the metaphor underlying "bias" in its original meaning.
Interesting article for me to stumble on today, as I've been thinking a lot about this dichotomy recently in my personal life and in observing my friends' problems.
Imagine the perfect narcissist. This person has no awareness of his surroundings - he has substituted any real observations for his own internal world. He has no awareness of the emotional states of others, obviously. He does not even have any awareness of his own emotional state.
Such a person could not function in the world. People with narcissistic personality disorder have to be getting in some data - in fact, a lot of the symptoms like rage wouldn't be possible if they weren't. These symptoms occur when the real data contradicts the narcissist's narrative.
Without an external data feed, you absolutely cannot grow as a person. And the process of taking external data, which doesn't give a crap about you, and bending it to your own context and your own needs, is a little selfish.
If you didn't do this, there'd be no difference between you and an insect. Stimulus->Response, with absolutely no middleman. But I'd argue there's a scary other end to that spectrum where your internal labyrinth entirely overwhelms experiential data, and that you can find most mental illness over there.
As a culture, I think we've overvalued our internal world. And maybe the solution to it is to mindfully increase our incoming data bandwidth - make a conscious effort, whenever you think of it, to fight the urge to retreat into your own mind and instead focus on sensation directly. Intentionally give more weight to the sensory over the analytical, the gestalt intuition over the carefully though-out, and the immediate over the future or past.
In other words, stop being rationalist, for a little while each day, and just do some things having no idea whether or not they're good ideas.
"seems to create permanent weird false beliefs for some reason"
Maybe there are multiple layers of "trapped priors". As we grow up we develop some base trapped priors that help us stay relatively normal (using the assumptions of the people around us). While psychedelics can help break higher level trapped priors, they also might break some of the lower level trapped priors, allowing the individuals to explore ideas further outside the norm. This can lead them to eventually redeveloping trapped priors that are far outside the norm, making them "weird"
> Although I don't have any formal evidence for this, I suspect that these are honest beliefs; the rich people aren't just pretending to believe that in order to trick you into voting for it. I don't consider the idea of bias as trapped priors to account for this third type of bias at all; it might relate in some way that I don't understand, or it may happen through a totally different process.
Seems like it's probably the same mechanism. You flinch from things that can hurt you. That can be emotional pain (like traumatic experiences), but it can also be expected future pain. "This argument will cost me down the line" is a type of pain to your subconscious, and it seems plausible that it'd have similar defenses.
I feel somewhat that the trapped prior here is the assumption that *what people are doing* when they have intercourse with others is gathering information for the purpose of critiquing their own points of view. But surely that is not obvious. My hypothesis in general is that almost all exchange of information between people is in fact for the purposes of persuasion and/or reinforcement of existing points of view. Id est, people quite rarely converse for the purpose of finding out something new, but much more often in order to reinforce something old (or just for social signaling that sorts other people according to their (old) beliefs).
To take an extreme example, the purpose of a wartime Department of Propaganda reading enemy newspapers is *not* to evaluate whether they should actually be an enemy, but to find new ways to defeat them. Even learning about the enemy's strong points -- at which he is good, noble, and true -- only serves as grist for the propaganda mill: knowing how and where he is strong, we can devise better attacks and defenses. Indeed, knowing the enemy is strong might very well *increase* our devotion to our own cause, inasmuch as we realize victory hinges on a fiercer dedication.
I wouldn't say as a rule we are trying to destroy each other, but there is probably a strong ongoing competitive ecology of social (and even personal) memes among us tribal primates, and it seems not unlikely quite a lot of our intercourse with others represents mere "foreign intelligence" and/or "propaganda" work in service of competing memes.
If that's the case, then we would naturally be confounded by the effect of evidence on beliefs, because we would have the mistaken believe that the mental "engine" at work has as its main purpose the accurate weight of evidence.
I thought it was Napoleon who said "Reasonable men have always a tendency to believe that others are like them, in which point of view they are not reasonable."
It's really too bad the "Rationalists" are unwilling to apply the irrational dog-phobia to their own irrational Trump-phobia:
<i>"Less-self-aware patients will find their prior coloring every aspect of their interaction with the dog. Joyfully pouncing over to get a headpat gets interpreted as a vicious lunge; a whine at not being played with gets interpreted as a murderous growl, and so on. This sort of patient will leave the room saying 'the dog came this close to attacking me, I knew all dogs were dangerous!'"</i>
Some are so non-self-aware they will even admit Trump did not say certain words ... yet still their Trump-phobia makes them CERTAIN that Trump means what he did not say. When Trump says "peaceful protest" they hear "violence and riot".
Since reading The Passion of Ayn Rand, by Barbara, the cheated on wife of Nathaniel who was with Ayn, it was clear that even smart folk, maybe especially smart folk, can rationalize whatever untruth they want to believe. Of course, if they hear "violence" when Trump is saying "peaceful protest", it's something different than mere rationalization. This phobia note explains it.
I suspect this is a semi-widely shared anecdote: one effective antidote to political trapped priors seem to be genuine interpersonal connections. Having discussions with friends you otherwise respect can make you interpret their ideas more charitably than coming from a stranger, and raise your threshold for the mistake theory --> conflict theory trigger. Not 100% sure if either part of this fit in the perception or context part of the model.
Deeyah Khan's white supremacy documentary shows this quite well imo, where a group of white supremacists grew found of her who has middle-eastern parents, through personal interactions, to the point where she (accidentally?) deradicalised a few of them. Many such cases in history ofc. I believe this was also Christian Picciolini's major point.
I think that at least to some extent, the problem with generalizing the scary-dog-friendly-dog example to a policy-evaluation case is that frequently, the "evidence" is not pure in the first place. For anyone but an extremely deeply cynophobic person, it would be harder to interpret more (as-per-non-cynophobe) friendly dog behavior as aggressive/scary, however narrow the evidence band.
What I mean by that is that when the absolute majority of people want to find out what "the science" says about X, they're not actually going to go and read papers - they'll read what their trusted journalists and/or bloggers say the science says, which will be partisan/skewed anyway. Even if they find a source that is very committed to not skewing research, reporting will rely on dumbing down the study to some extent, which inevitably introduces the risk of simplifying the conclusions into something they are not saying.
I wonder how effective it'd be to take someone with trapped political priors completely out of their environment -- both geographical and virtual, to get them away from their friend circles and echo chambers. Being transplanted into a different community would not only increase the evidence they're exposed to for alternative views, but there'll also be a social element to consider. Throwaway comments that are completely based on everyone's collective priors (and evidence) will no longer be welcome, and will be challenged.
As someone who grew up in a highly conservative and religious area and then moved away, I can vouch that some entrenched priors were removed, though it took years in some cases. It's obviously a large undertaking and wouldn't happen unless the person involved is wanting a life change, or up for extreme experimentation. And of course, many of these folks with trapped priors won't *want* to be transplanted into a different community. Nevertheless, if those hurdles are overcome, it could accomplish the goal of bringing the trapped priors back from the "threshold"?
I think the "Point of No Return" is relative to not just the "amount of evidence you'll be exposed to" but more importantly the amount of SAFE evidence you're exposed to - an individual "zone of safety" that constrains what defines evidence.
I think if you have someone who knows what a trapped prior is, and recognizes yours, and they're someone you trust, you can actually get out of a trapped prior pretty easily. As you mentioned: slow and gradual exposure in safe doses.
For something like a phobia, this seems relatively easy to provide. I had a friend significantly reduce my phobia of dogs just by teaching me a few things about dog body language using pictures and videos. I'm still not great around them, but it's a lot easier to deal with them when I need to. I think it was maybe an hour of her time, total?
In particular, she mentioned that a lot of "hostile" dog body language in movies is actually a friendly dog visual, and then angry growling is dubbed in - handling an angry dog during filming is much harder than just using a generic growl! She used a few examples and fixed that prior of mine, and now I can at least tell whether a threat is real or just me having a panic attack.
For other domains, like politics, it's probably a lot harder to develop that sort of trust. But just understanding that some people out there have trapped priors should improve your ability to cut short unproductive debates.
Now I understand that I am probably doomed in covid issues. That's interesting that a year ago I thought myself moderate, I warily supported restrictions though was suspicious of lockdowns.
After five months of being locked up in a tiny apartment and then an escape from Chile (where I lived for five years) to Russia (where I was born and lived before emigration) I became doomed exactly in a terms described by Scott. Now when I hear somebody tells something like "Let's wear masks, this prevents more hard measures" I can't help but think that person is either extremely naive or evil. In Chile people were mostly mask-compliant and they quickly got a long lockdown that is still partially in action (and it is summer there). In Russia many people tried their best not to wear masks, (some people were beaten, at least one killed after they insisted, that others should wear masks) - and there was a short lockdown in Moscow, that ended in a month and no lockdowns after that. Last November an official from my city (Saint Petersburg) said "If people do not wear masks they will not stay at home, so no lockdown". And the same trend you can see through all the Europe and America. So I think every person who enthusiastically wears a mask objectively increases the chance of a new strict lockdown.
I confess I have suffered from trapped priors. What helped me was engaging in an activity that forced me to often and consistently update my beliefs based on empirical data: trading. This somehow loosened up my trapped prior, as if my whole "system" had become more "open". I believe other activities could have a similar effect. Building something, craftmanship, well I guess anything non-theoretical could work. But 100% agree this is a hugely important field to study.
Practical application of Bayesian statistics often involves probabilities so small that the only way to work with them is on a log scale. Enough very weak evidence over time can overcome a badly specified prior that giving this sort of tiny prior probability to the truth. On a log scale, evidence keeps adding up.
"Ironically, my prior on this theory is trapped - everything I read makes me more and more convinced it is true and important. I look forward to getting outside perspectives."
Your deep dives into the ultimate question of perception, cognition, and everything, when considered together as a series, are beginning to look kabbalistic.
I'm not sure how much rationality as a cognitive exercise has to offer for this particular issue. This idea of the Bayesian Brain is really helpful and informative for conceptualizing a lot of things (I think this paper really influenced that idea: https://www.psy.gla.ac.uk/~martinl/Assets/MCMPS/KnillPouget04.pdf). But biology doesn't actually care about truth, and your brain was not evolved to process information and find truth. That is mostly a happy by-product of the fact that a world model is generally more useful for survive if it corresponds with truth, but that might not always be the case. I think expecting everything to be Bayesian just because some things are is in itself a trap.
Particularly for things directly related to survival (food, safety, sex, group membership) it seems likely that your brain just isn't going to be prioritizing objectivity, maybe to the point where objectivity is physically impossible based on whatever biological implementation these have. If this is true then no amount of information can ever change this sort of belief. You would need to tap into whatever incentive structure created it in the first place.
You didn't mention the role of heros, idols, experts, cult leaders, ideologues etc.
In the case of a phobia, it's not relevant, but for ideological fixation, a trusted expert can go a long way.
Most people don't think for themselves. They set a sort of network of trusted experts and then go with the flow.
These respected allies provide ongoing confirmation that the cult is on the right track.
"If these people, whom I trust, all believe X, then I believe X. If you show me evidence to the contrary, it is necessarily false, however convincing it might seem superficially."
If the cult leader sits everyone down and says hey, it was all a lie (with lots of details)... perhaps some people might stick with the lie but there's a much better chance that many cult members will be released.
The problem then becomes networked falsehoods spread by major influencers. Still a big problem, but not the one you're describing.
Is anybody else upset at the likes being gone? Now I can't distinguish the popular comments from the unpopular ones, which hurts my ability to read the thread.
Perhaps a summary is that priors can “update” input signals before those transformed inputs are used to update the prior. Trapped priors occur when the input signals are always transformed such that they subsequently confirm the prior. This can actually be rational behavior if the reward experienced for having the prior confirmed is sufficiently large.
Well, this is #549 for “Trapped Priors As A Basic Problem Of Rationality” : What a loaded title to unleash a torrent of comment.
Humans exist in one physical universe but live in the multi-verses of the mind created by the brain with sensory input, connected by neurologic (somatic and autonomic nervous system), and actuated through muscular systems. Every aspect of these systems exhibit a distribution which influences capabilities, behaviors, and abilities etc which vary in time. Terms such a “standard observer”, “neuro-typical”, normal, abnormal, sane, crazy are used to characterize populations. These latter terms are particularly important to Scott’s primary profession.
Scott characterizes the learning process of the brain’s neural network as “the brain combines raw experience (eg sensations, memories) with context (eg priors, expectations, other related sensations and memories) to produce perceptions. You don’t notice this process; you are only able to consciously register the final perception, which feels exactly like raw experience.” For some reason he prefers the term priors to history and perceptions to thoughts.
As the study of arguments that are correct in virtue of their form, logic is of fundamental importance in the study of rationality. The study of rationality in logic is more concerned with epistemic rationality, that is, attaining beliefs in a rational manner, than instrumental rationality.
Economics: Rationality plays a key role in economics and there are several strands to this. Firstly, there is the concept of instrumentality—basically the idea that people and organizations are instrumentally rational—that is, adopt the best actions to achieve their goals. Secondly, there is an axiomatic concept that rationality is a matter of being logically consistent within your preferences and beliefs. Thirdly, people have focused on the accuracy of beliefs and full use of information—in this view, a person who is not rational has beliefs that don't fully use the information they have.
Debates within economic sociology also arise as to whether or not people or organizations are \"really\" rational, as well as whether it makes sense to model them as such in formal models. Some have argued that a kind of bounded rationality makes more sense for such models.
Others think that any kind of rationality along the lines of rational choice theory is a useless concept for understanding human behavior; the term homo economicus (economic man: the imaginary man being assumed in economic models who is logically consistent but amoral) was coined largely in honor of this view. Behavioral economics aims to account for economic actors as they actually are, allowing for psychological biases, rather than assuming idealized instrumental rationality.
Artificial intelligence:” Within artificial intelligence, a rational agent is typically one that maximizes its expected utility, given its current knowledge. Utility is the usefulness of the consequences of its actions. The utility function is arbitrarily defined by the designer, but should be a function of \"performance\", which is the directly measurable consequences, such as winning or losing money. In order to make a safe agent that plays defensively, a nonlinear function of performance is often desired, so that the reward for winning is lower than the punishment for losing. An agent might be rational within its own problem area, but finding the rational decision for arbitrarily complex problems is not practically possible. The rationality of human thought is a key problem in the psychology of reasoning.
As an example, consider the work on autonomous vehicles which use SLAM: Simultaneous localization and mapping.
By analogy, humans navigate in their multi-verse for numerous purposes: staying alive, reproduction, among others. Reasoning and rationality likely evolved in humans because in a Bayesian analysis they are traits that lead to positive outcomes such as staying alive, reproduction, among others.
As an example on distribution, human vision has color, focus sharpness, response time, spectral sensitivity vs light intensity etc.
In the United States, about 7 percent of the male population—or about 10.5 million men—and 0.4 percent of the female population either cannot distinguish red from green, or see red and green differently from how others do. More than 95 percent of all variations in human color vision involve the red and green receptors in male eyes. It is very rare for males or females to be “blind” to the blue end of the spectrum.
The 8% of colour blind men can be divided approximately into 1% deuteranopes, 1% protanopes, 1% protanomalous and 5% deuteranomalous. Approximately half of colour blind people will have a mild anomalous deficiency, the other 50% have moderate or severe anomalous conditions. { https://iristech.co/statistics/ } Women tend to have enhanced blue sensitivity compared to men.
This is certainly consistent with my work as a former military physician on the problem of PTSD from a neurologic (versus cognitive/psychiatric) perspective. Generally the best results when people heal are from eliminating and resolving "trapped priors", which I have seen most commonly (although not always) related to unprocessed shock-level events. I don't find therapeutic value in treating the behavior patterns or trapped priors as independent entities from the experience ... it simply reinforces the avoidance which is a feature of the trapped prior itself.
I have seen this resolution happen routinely by the individual coming into the memory from the outside (i.e. what would you say to encourage that person). This creates a positive association, instead of the reinforced negative of the dog perception example. This could be likened to putting lotion on a burn and has more quality control over the experience ... it's still a burn, but the experience was fundamentally soothing and not irritating, so the relationship with the burn changes (to one of attraction to soothing/love that heals instead of avoidance of pain).
Since most hyperpriors are established by especially important/intense or repeated events, things that stabilize the brainstem function and signal survival are usually helpful - basic breathing and nursing movements provide a bottom-up sensory stimulus of safety that can be accompanied by a top-down interpretation to build a stronger experience ... but it still has to be connected back to the point of unprocessed experience or you will be fighting a long time!
The problem with attempting to overcome priors without resolving the causative events (i.e. reason or stimulate your way out of them without interacting with the primary event) is that you still have unresolved/unprocessed sensory data that will continue to inductively reinforce the prior - strong, unprocessed sensory experiences still hurt (i.e. unprocessed) and serve like an infection which will continue to result in future sensory suppression (which perpetuates the problem) until resolved.
The other challenge is that these are short-term protective (against pain), and so you are fighting against a survival/protective mechanism - that some evidence suggest may relate to physiologic vasoconstriction of focal areas of the body and brain, implying potential therapeutic value in understanding circulatory mechanics.
But on the survival side, as Vincent Filleti (ACES work) says about maladaptive behaviors (substance use, etc ...), "you are dealing with people's solutions, not their problems". We view them as problems, but they view them as physiologic solutions for pain, and that has to be understood in terms of why these are so hard to simply eliminate with a snap of the fingers!
Thanks for the great summary and post! Very stimulating and consistent with my "perception and experience thus far" of the world :) Biases successfully reinforced. Great day!
> The basic idea of a trapped prior is purely epistemic. It can happen (in theory) even in someone who doesn't feel emotions at all. If you gather sufficient evidence that there are no polar bears near you, and your algorithm for combining prior with new experience is just a little off, then you can end up rejecting all apparent evidence of polar bears as fake, and trapping your anti-polar-bear prior. This happens without any emotional component.
I can see that in theory this could happen without emotions, but I’m wondering how frequently we see practical examples (in humans) where the trapped prior isn’t tied to an emotional response.
It seems the cases shown here include:
1. Clearly emotionally-charged subjects in politics, where it’s pretty easy to see the “disgust” / “fear” response flooding the perception and preventing a correct update of priors.
2. Potentially less-clearly emotional cases like tribal / religious identity (cool-headed analysis of political questions that nonetheless fall prey to bias? Religious cults?). Do we want to classify these as not emotional? I could see a case either way. Tribal identity seems like a pretty deeply-rooted part of how the human brain processes social information/situations. And the outgroup responses seem to tie in to emotions at a pretty low level. But I could see an argument that this is distinct machinery from lizard-brain fear/disgust responses.
3. Clear pure-epistemic cases, where there is no emotional flooding blocking the updating of priors. I’m unclear if this set is empty. Is flat-earth-ism one of these? Someone gets infected with a meme that injects the belief “The conspiracy is going to try to convince you that I’m wrong with scientific-sounding facts. If they sound science-y and come from people in lab coats that will be proof that this belief is correct”. Now without any emotions, the update mechanism is subverted. Perhaps we put the cognitive bias of attending more to facts that agree with our previous beliefs here?
> But the context is a very strong prior that dogs are terrifying. If the prior is strong enough, it overwhelms the real experience.
While this is described as the “pure cognitive / no emotion” case, this sounds more like 1, strong emotional flooding. Even without lowering the bandwidth of the present-experience through the trauma response, if the fear association is strong enough it floods/drowns out the “objective” / external experience.
Maybe 3. is actually a different mechanism; 1/2 could be purely explained by emotional flooding causing the subjective experience to be negative, therefore priors aren’t updated. 3. could have something to do with the non-Bayesian nature of the human updating machinery that Scott mentioned in the previous article:
> it can't possibly be this simple, maybe "bad" and "not bad" are binary states and once your brain decides something is bad it fails to process that it was slightly less bad than expected
If “binary” seems too simplistic, it could be some sort of quantization or even just a highly non-linear scale where input of -0.9 is scaled to -0.99999 which would require more than a lifetime of updates to shift meaningfully.
In summary I think this is a useful concept, but there could be two fairly distinct mechanisms at play here that are both causing the overall family of “trapped prior” update failures. In particular it’s not obvious to me that the rationalist experiencing an emotional flood should respond in the same way as a purely epistemic case where cognitive biases are preventing updates from happening properly.
Or as we used to say, One person's modus ponens is another's modus tollens. (Didn't need any of that fancy real-number probability stuff, and we liked it.)
“I'm not sure what level of evidence could possibly convince them.” reminded me of the “arguments” trying to disprove something by “if this were true, some proof would have been found somewhere” but missing the point that for the proponents, the proofs _had been_ found. It’s just the opponents are ignoring the “proofs” as obviously nonsense/crackpot/irrelevant/…
This is exactly what is happening with people who are excessively fearful of COVID (ie, believe that COVID is worse than a once-per-decade flu):
- self-serving bias: I can work remotely and I am saving more money that ever. My life is calmer. Therefore, the economic damage to poorer people is similar, and I don't have to expend too much mental energy drilling down into the question.
- "more scientifically literate people are more likely to have partisan positions on science". In this case (as in the case of eugenics in early 1900s), they "follow the science" without thinking to critically.
"The other promising source of hope is psychedelics. These probably decrease the relative weight given to priors by agonizing 5-HT2A receptors. I used to be confused about why this effect of psychedelics could produce lasting change (permanently treat trauma, help people come to realizations that they agreed with even when psychedelics wore off). I now think this is because they can loosen a trapped prior, causing it to become untrapped, and causing the evidence that you've been building up over however many years to suddenly register and to update it all at once (this might be that “simulated annealing” thing everyone keeps talking about. I can't unreservedly recommend this as a pro-rationality intervention, because it also seems to create permanent weird false beliefs for some reason, but I think it's a foundation that someone might be able to build upon."
If I'm understanding this correctly, I don't think the "for some reason" part is actually mysterious - if a drug makes you more heavily weight actual experiences over priors, and also gives you weird actual experiences uncorrelated with reality, then it makes sense that you'll both be more likely to update on the true evidence you've been collecting and ignoring in your everyday life AND that you'll also be more likely to update on the false evidence the drugs are feeding you.
More generally it seems like this post takes the implicit view that generally people will tend to have overly strong priors and can improve their reasoning ability by weakening them; this may well be true at the population level, but I'm pretty sure it's also possible for one's priors to be too weak. (Which may look like e.g. taking supernatural explanations too seriously when you did not already believe them. (As opposed to a trapped prior in favor of a supernatural explanation, which is common too.))
Maybe this is why there seems to be a rationalists-->[people who are into woo and don't seem to make a ton of sense or be especially good at reasoning] pipeline? For a while weakening your priors has good effects on one's reasoning ability compared to baseline, and then as some point it has bad effects instead?
This is plausible, but let me give a somewhat different theory (as always, the truth may well involve both).
Let's consider the standard model Level-Public explanation of our times:
person did <X> because of <reason couched in terms of ideology>
The salient feature here is the *ideology*. 200 years ago it was Christianity, 50 years ago it was Communism, today it's Woke. The point, in each case, is not that there's a basic moral intuition
- "do unto others as would be done unto you"
- "pay workers a fair wage"
- "treat women like men"
it's that there is a massive collection of deductions as to how the world should be, excuses for why the world does behave the way the model says it does, and a closed off worldview such that the ideological worldview is always able to triumph over "truth", regardless of how truth is demonstrated.
OK, now let's consider a level-N+1 explanation of the prevalence of ideology.
Consider these two statements:
- "the Nazis executed a very clever tactic during the invasion of France whereby they deliberately bombed towns and villages of no military value so as to create streams of refugees that clogged the roads and blocked Allied transport to the front"
- "Ivanka Trump is substantially above average in physical attractiveness"
My experience is that most people (at least, let's say, Democrats for the second) have a real problem with these statements. Their minds rebel at the concept of something being "good" along one axis and simultaneously "bad" along an orthogonal axis, and most people prefer to resolve this by simply asserting that there is no such tension -- the Nazis were not clever, they were simply evil; Ivanka is not pretty, she's also just evil.
Think I'm being ridiculous? Look at the response to Karlheinz Stockhausen's 9/11 comments, eg
This n+1 level is a superset of the public level, because it obviously explains the public level ("truth" or <some argument> is good along one axis, but bad [disagrees with the ideology] along an orthogonal axis; let's resolve the tension by picking ideology over truth/<good argument>)
But this explanation is more powerful because it covers cases of tension (like the Nazi case, as opposed to the Ivanka case) that are not objects of active ideological battle right now - few are generating their opinions about Nazis by considering "what have my tribal leaders said about this, what have the tribal opponents said, let's synthesize a response that maximizes the first and minimizes the second")
But we can, I think, do even better. What's we've concluded so far is that most people can't sustain a multi-dimensional "stance" towards something; they feel they have to collapse all the dimensions down to a single-dimensional "good vs bad" continuum (and mostly not even a continuum, two clusters, one of good, one of bad). So why should that be?
I suspect the underlying driving factor is some sort of tribalism inherited from the very earliest social species days. Things were judged in terms of "good for me/the tribe" (and usually, like GM and the US, those were synonymous) vs "bad for me/the tribe". The subtlest this got was maybe something like "Yes, they are our enemies and we want to kill their men. But while we're there we might as well rape their women, always good to have a backup copy of the genes floating around."
This gets us to essentially the same place as Scott, but the mechanism (most minds collapse all value judgements to a single dimension) and the reason for that (ancient tribalism) are substantially different from Scott's explanation (their Bayesian machinery has broken down).
I think my explanation is more powerful than Scott's (though his mechanism may be a part of how my larger mechanism works) because Scott's machinery doesn't really have a story for the sort of example I gave for clever Nazi's, cases where people don't want to admit a good vs bad tension, but this is not really part of living in a bubble/locking one's priors to absolute truth.
(And such cases are common! There are similar very clever financial manipulations [that hurt a lot of innocent people], or beautiful art works by horrible people, or very clever design elements in <chip/operating system from Apple/Google/MS/Intel/Facebook/whichever tech giant you choose to hate>, or elegance in the design of a fusion weapon.)
Step 1: first we reinvent Freud's theory of repression:
> Van der Bergh et al suggest that when experience is too intolerable, your brain will decrease bandwidth on the "raw experience" channel to protect you from the traumatic emotions.
Step 2: then we pull from one of Freud's favorite influences, Spinoza, to reinvent his "associative" theory of emotions:
> you're in an abusive or otherwise terrible relationship. Your partner has given you ample reason to hate them. But now you don't just hate them when they abuse you. Now even something as seemingly innocent as seeing them eating crackers makes you actively angry.
Step 3: then we reinvent Zizek's theory of ideology, but without the parts that specifically focus on how it's specifically *symbols* that people tie their deep priors to (i.e., you can tell a Republican about a program that Democrats would support, and they will be in support of it so long as you don't tell them that it's a *Democrat* program).
> zealots' priors determine what information they pay attention to, then distorts their judgment of that information.
There's also a subtle normative/ethical stance in Scott's post, by implying that some political stances are more "detached" from "reality" than others, i.e. that in an entirely symbolic game, some people are making "wrong" or "bad" decisions which we need to "fix".
Toss in some REBUS, and we're back in MK-ULTRA territory, trying to use psychopharmacology to "fix" people's beliefs, a program that worked "very well":
> Its goal would be a relatively tractable way to induce a low-prior state with minimal risk of psychosis or permanent fixation of weird beliefs, and then to encourage people to enter that state before reasoning in domains where they are likely to be heavily biased.
Finally this:
> Tentatively they’re probably not too closely related, since very neurotic people can sometimes reason very clearly and vice versa, but I don't think we yet have a good understanding of why this should be.
My stance is that they're absolutely related. Think of a "trapped prior" (i.e. a traumatic kernel, in Freud's language) as a "black hole" that "warps mental space-time" for the individual. When you're traveling around (thinking) far from the black hole, then the thoughts "make sense", but the closer you get to the black hole, the more things get distorted, irrational-seeming.
The problem is, as Lacan says:
> there is no position of 'mental health' which could be called 'normal'. The normal structure, in the sense of that which is found in the statistical majority of the population, is neurosis, and 'mental health' is an illusory ideal of wholeness which can never be attained...
So space-time will form black holes, the only question is where are they located. Perhaps you want people's mental space-time to warp in more familiar places ("my bias is actually the normal"), and want to establish an ethical program to ensure that more people are "like me".
This wouldn't be the first time that institutions or groups have embarked on such a program, but I urge you to think through the ethical implications, if "eliminating bias" no longer has a "privileged" position with respect to "reality". One thread to pull on: what happens to the ecosystem of thought and communication if we successfully eradicate "diversity of bias"? And what will the material consequences be?
I'd like to take a different lesson from the psychedelics research. They focus a lot on 'set' and 'setting'. This is the idea that it matters how you enter into and in what context you experience a trip. They focus a lot on this, and I've heard some insist that it's entirely the difference between a good and a bad trip.
I've never used them myself, but I hear a lot of descriptions of psychedelics as effectively eliminating the 'prior' side of the models above, leaving only sensory experience as an input.
There's a place we can build this into Scott's model. He brought up the idea of weighting priors/sensory evidence, but never asked the question, "What's the weighting function?" Yet from his desensitization example, as well as the discussion about emotion-motivated setting of priors, we can make the following guesses:
- Priors are weighted by the emotional context under which they were established
- Sensory evidence is weighted by the set/setting under which it is experienced
This is why the gradual approach of pictures of puppies works well, but the locked room doesn't. It's not that priors can't be reset, it's that the sensory evidence that should reset them only gets sufficient weight when amplified by the correct modulator (set and setting).
Andrew Huberman often refers to the cognitive regulation of emotion through eye movement - there is evidence that a subject recounting a traumatic event while moving the eyes laterally extinguishes the fear or distress. This kind of study: https://www.jneurosci.org/content/38/40/8694
> this might be that “simulated annealing” thing everyone keeps talking about
Umm, who's talking about this? I feel deeply out of the loop, but google is just returning results from computer science. Is there a fad among rationalists of trying to overcome biases by believing obviously-false claims to see if they turn out to be true, and they're calling it "simulated annealing," and I haven't heard about it?
Some thoughts based on the observation that computational power (both operations and data storage capacity) isn't free.
*****
Speculation: the writeback of the updated prior is a nontrivial portion of the total energy cost of a Bayesian update in humans.
If so, then one obvious optimization from an overall-energy-cost point of view is to only do a writeback if the prior has changed by a non-negligible amount.
...and then you end up in the sorts of trapped priors that you describe.
*****
One other related observation: you end up with exactly the same sort of thing with deterministic rounding modes and restricted-and-finite bitwidths for the prior, such as you would end up with if you were trying to minimize storage requirements.
(If the rounding was _non_deterministic and the bitwidth small then you could end up in a funny scenario where small amounts of progress generally wouldn't happen - you'd see a nontrivial amount of progress or none. I don't know enough to know if this sort of thing happens.)
I love this piece. An assumption of Bayes Rule is that priors do not influence new evidence, but obviously they do. If magnitude but not direction of evidence is altered, then we still converge on truth. If direction is altered, we have "trapped priors".
Maybe we can explain the trap if there is a prior on the expected physiological reaction that constantly underestimates the physiological reaction because it is tainted by some rational expectation. My rational brain tells me that being with a dog will be ok. But my physiological reaction turns out to be quite bad for no rational reason. That might reinforce the physiological reaction (the reaction was actually bad and much worse than I expected overall) and it may keep the rational expectation unchanged because come on, puppies. No clue whether rational vs physiological priors are a real thing but it seems hard to escape.
Yet, as Stephen Senn persuasively argues, "the placebo effect is largely a myth." Perhaps a more profitable approach would be to figure out how so many people avoid falling for illusions.
I used paint's color picker to check if the raw pixel data at the chess pieces in both the images are grey. I tried to avoid the smoke that was hard. https://imgur.com/avQC6Kx I think the white pieces are actually grey and look whiter than it actually is. But the black pieces are actually black not grey. I think it would be better if you could post a 3rd image without the smoke.
I think perhaps "bitch eating crackers" is a poor example, because other people eating crackers is mildy annoying. If a stranger were eating crackers opposite me on the train, I might be irritated by their lack of consideration. If a friend is eating crackers, I'm less likely to be bothered. In part, I think that's because I *could* ask my friend to stop if it was bothering me too much (whereas I would feel uncomfortable making that request of a stranger), and the mere fact that I could make the request puts the cracker noise within my domain of influence, rather than being something externally imposed.
With the toxic relationship, the problem is (I speculate) that the guy fears that if he asks his partner to stop eating crackers, that would trigger a massive row. In turn, the mere fear of that row, without his partner actually doing anything, makes him furious, because what sort of relationship is it where you can't even make a such a simple request? That might all be paranoia, in that the partner would actually happily refrain from eating crackers if she knew it was annoying, but it *might* be completely accurate. The partner really might understand a request to desist from cracker eating as hostile and might react negatively to that. If so, there really is a problem in the relationship. In that case the guy's problem isn't epistemic but pragmatic: he correctly identifies a problem in the relationship, but is unable to identify a solution and (perhaps) reacts in a way which only exacerbates the situation.
I don't have time to read all the comments, so I assume someone may have addressed this. But this isn't an issue with Bayesian processing in my view. If we assign a 0% probability to an outcome in our prior, then Bayesian updating will produce "trapped priors". This just highlights the fact that regardless of the data observed, the observer must be open minded about change.
In my experience, Socratic questions opens priors. But it is exhausting to pursue and takes forever.
No you don't. You have made that very explicit in our one on one conversation. You want to want to get outside your perspective, but you accuse me of witchcraft on the grounds that I have a historical track record of getting people to change their perspectives.
While this discussion is about phobias, traumas and negative priors, I wonder if it also works for some positive priors / emotional orientations toward things? ie whether having a massive crush on someone means that any time you interact with them, you get a hit of good feelings (or "they make you feel really good" as you may interpret it), which strengthens your prior that they're genuinely wonderful, which means next time you see them they make you feel great again... people do seem to get temporarily trapped here, but it doesn't entrench the way phobias do. Maybe because our brains are a lot more reactive to single bad things than to single good things?
A recent paper tried to replicate the Fernbach et al findings referred to above, and found that they didn't replicate.
"Asking People to Explain Complex Policies Does Not Increase Political Moderation: Three Preregistered Failures to Closely Replicate Fernbach, Rogers, Fox, and Sloman’s (2013) Findings
Fernbach et al. (2013) found that political extremism and partisan in-group favoritism can be reduced by asking people to provide mechanistic explanations for complex policies, thus making their lack of procedural-policy knowledge salient. Given the practical importance of these findings, we conducted two preregistered close replications of Fernbach et al.’s Experiment 2 (Replication 1a: N = 306; Replication 1b: N = 405) and preregistered close and conceptual replications of Fernbach et al.’s Experiment 3 (Replication 2: N = 343). None of the key effects were statistically significant, and only one survived a small-telescopes analysis. Although participants reported less policy understanding after providing mechanistic policy explanations, policy-position extremity and in-group favoritism were unaffected. That said, well-established findings that providing justifications for prior beliefs strengthens those beliefs, and well-established findings of in-group favoritism, were replicated. These findings suggest that providing mechanistic explanations increases people’s recognition of their ignorance but is unlikely to increase their political moderation, at least under these conditions."
There's an interesting discussion on this issue in E. T. Jaynes' "Probability Theory: The Logic of Science". Section titled "Converging and Diverging Views" on p113. If you like Bayesian probability theory and rationality, you really will enjoy this book.
I suspect our mind pulls a simple but effective trick on us. When faced with new sensations that run counter to our strong prior, we actively remember past sensations (scary dog memory, all of my old arguments against capital punishment). But we _falsely_ interpret this rehashing of old facts as new input. Repetition of already known facts is an effective tool of persuasion even if it should have zero value from Bayesian point of view. And we do it to ourselves. So our strong prior is in effect updated with weak evidence against and strong (echo) evidence for.
You might see this play out when reading an oped you disagree with. Rather than only interpreting the new input and judging it as unconvincing, you’re probably pulling out some of your favorite counter-arguments and (if you’re anything like me) repeating them loudly to anyone unfortunate enough to be within earshot. Being already known, they should not update your prior. But I suspect they actually do.
Something that doesn't make sense to me about this model: If VdB et al suggest that experience is decreased to protect from negative emotions, that suggests that emotion belongs to 'sensation' or the empirical input of experience. But surely, 'scary' refers to the emotion of fear ('dangerous' would be the emotionless empirical equivalent), yet here the model depends on a prior of a scary, not a dangerous dog, suggesting emotion belongs instead to the predictive 'prior' input to experience.
Also, it just seems obvious that people can and do deliberately eat crackers in a bitchy way.
Look forward to advertising and PR agencies getting their hands on that research that shows a reliable method for releasing trapped priors. Every aspiring cult leader will want to make use of that data.
"The biggest problem I used to have is people used to have intractable biases against buying the snake oil I was trying to sell them."
The fundamental error of this approach is the need to explain biases in the first place. Unbiased ness is suboptimal in finite sample inference. Before you start looking for explanations , first you need to make sure you have the proper model of optimal cognition.
I would recommend using a dial in the center of the gray box to represent weighting. That way you can still use your restricted pipe metaphor while also capturing what the color output would be normally.
Just a note, that "wine study" was done on enology students, not sommeliers.
(A nice writeup on the myth)
https://sciencesnopes.blogspot.com/2013/05/about-that-wine-experiment.html
The study has be telephoned to death by the popular media thanks in no small part to the snobby gatekeeping mentality of the wine industry, whom people just want to say are full of bull. (I say this as some one with an amateur interest in beer, wine, and spirits. Someone with a decent palate who regularly does blind tastings)
This analysis is exactly correct, IMHO (or my trapped priors lead me to think so, anyway....) I think the fundamental question, unaddressed here, is "How do you catalyze people into recognizing, accepting, and acting on the fact that they have a trapped prior?" Anyone? Bueller?
The mechanism doesn't seem like it has to lead to irrational behavior 100% of the time. If it happens to correspond accurately to an expected value assessment (dog bites are unlikely, but really really bad; believing dogs are bad prevents dog bites; dog phobia is justified from an expected value standpoint) it might be functionally valid even if it's epistemologically invalid. Rational behavior is more important than rational belief, and there's no reason they have to correspond.
The analysis here seems correct and incredibly important. If we could get people to recognize this concept of trapped priors within their own experience, maybe it could help us break out of this emotionally charged era and lead to greater cooperation and less conflict.
The problem as it appears to me is, how do we communicate this idea to the public? The way you've presented it here works well for communicating with people who tend to view concepts in rigidly logical ways, but I think this approach will fail with the general public. I'm thinking of my wife, who is a therapist who has done a lot of work with trauma. She has said essentially the same things presented here in our conversations, but she frames it in a way that is rooted almost entirely in personal experience. If I were to try to bring up the concept of trapped priors this way I don't know that it would click with her, or even if she did agree with the theory she might reject this way of thinking about it as unhelpful to her clients with trauma.
So how might we present this in a way that is easily digestible?
Maybe other people have already thought about this and found ways to present it to a general audience. I'd be interested to know if anyone has resources like that. I think the inability of science to communicate its results effectively is one of the greatest barriers to progress.
Brings to mind a conversation I had with a politically radical relative, whose extremism on politics was noticeably ratcheting up at the time. I shared with him a study about how easy it is to get partisans to agree with false statements by presenting them in an appropriately partisan context. He responded by explaining at length how the explicitly false statements on his side of the issue were actually *true* and the study just proved even more how right his political beliefs were.
Very educational conversation for me... Changed my view of political belief a lot.
I definitely notice my strong prior against assertions made by certain parts of the political spectrum, but I'm not sure how much to fight that - if my prior is "this person or group has lied or misled in a large portion of the things I've heard them say" that seems like a useful, rational thing to allow into my judgement of new assertions. Or maybe I have a stuck prior which leads me to misinterpret the things they say, thus making them seem to be liars? Bit of a rabbit hole.
FWIW, some people think confirmation bias (i.e. interpreting seemingly unfavorable evidence in a way that favors your priors *and* almost exclusively searching for evidence that confirms your priors) is rational: https://www.kevindorst.com/stranger_apologies/confirmation-bias-as-avoiding-ambiguity.
The common thing that is often mentioned in the context of talking people out of an epistemic "echo chamber" or out of a radicalist cult, or even just out of garden-variety homophobia, is getting to know a friendly person who is from the outside, and gradually coming to like them and eventually trust them and listen to them. This really sounds a lot like the desensitization therapy that works - it's never going to help if Ted Cruz or AOC themself come to get to know someone and show that they don't bite, but if someone who seems nice and isn't already processed as a political symbol gradually reveals their niceness while also gradually revealing their partisanship (or the fact that they're gay) then it can overcome the defense mechanism.
A cute example for me was this sentence, from your post:
"Your prior that they're bad has become trapped."
I couldn't look at this sentence without seeing a typo. The two words "your" and "they're" in close proximity told my brain that one of them must be wrong. Weird.
as far as the relationship with political polarization goes i think this is a good argument for policies which increase diversity. if you have a trapped prior about people of type x being exposed to lots of people of type x will be required to overwhelm your prior. american history x is a great dramatized version of this. or daryl davis' stories. so ubi/remote work so that urbanization decreases somewhat, or maybe even something like mandatory federal service.
There is another factor that comes into play when this relates to politics. For someone with cynophobia, the material they are being presented with is actual evidence: whether it is pictures or direct experience with a dog. But in politics, the vast majority of the discussions do not revolve around evidence, but on the reporting of evidence. Nobody outside masochistic bloggers and other scientists are reading the original scientific studies about epidemiology, sociology, climatology, etc. They are reading reports and summaries about the studies produced by think tanks, academic institutions, professional societies, politicians, or more often the media. So it is not evidence that a voter is being presented with, but reports from intermediaries. As long as everyone trusts those intermediaries to the same degree, then the argument holds, but if someone has priors about the agenda or trustworthiness of the intermediaries, then that becomes the relevant prior.
And it's not just the prior about trust in the reporting, it is the prior about what you do next. Let's say you are a staunch Republican who believe that climate change is an overblown issue and being used by the left to advance their political agenda. You hear about a report that climate change is a problem. Even if you think of yourself as a rational person who wants to keep an open mind, you look into that report, and find out that it was produced by a group of scientists at an Ivy League university in collaboration with scientists in Europe, and has already been trumpeted as a clear argument for banning fossil fuels by a liberal organization. So at this point, before you've even had a real chance to read a sober account of the report by, say a neutral expert that you trust (assuming that one exists), you've already had 5 or 6 different priors triggered, all reinforcing your skepticism.
It is not just the encounter with the evidence where you engage your prior, it is the other 10 encounters on the way to the evidence that already have associated priors. It's like a person with cynophobia being told they are going to be taken to a room with a cute puppy in a cage, but on the way to the room, you can hear dogs barking, smell the dogs, and see leashes and bones scattered on the floor. It is a really hard thing to then still walk into the room. To provide a clear path to consider evidence that had political implications, you need a trusted source to explain it; one that you can turn to before the dozen partisan - or perceived partisan - entities can yell you back into your priors.
As far as I know the current view on exposure therapy isn't only about gradual exposure. It is in addition about getting the patient to make specific predictions what he/she expects to happen during the exposure and to try observing this as closely as possible. This could be worth a try with political biases, too.
Have another stuck prior, from a book called _How to See Color and Paint It_, by an art teacher who would have his students cut windows in 3" x 5" cards so it would be possible to just look at small areas. Then he'd have them go to the bay at dusk and ask them what color are those (brick) building's across the water?
They'd say "red", and then he'd have them look through the 3" x 5" windows, and the buildings were actually blue in that light. The prior for how the buildings look in typical lighting overwhelmed the actual sensory experience.
My experience with cynophobia fell somewhere in the middle of the treatments. For most of my life, I've been really scared of all dogs. I can keep it under control by expending mental effort, but it's taxing. Then, a few years ago, the ceiling of my apartment collapsed and the landlord refused to fix it. My best recourse was to move in with a friend in the area. The only problem? Another person living with him had a dog, the most golden retriever golden retriever I can recall meeting. I'd encountered this dog before, and could tolerate it, but only with effort.
Still, with a choice between staying in a room with a collapsed roof or a house that had a dog, I chose the latter. Eventually, I stopped freaking out when the dog started barking as I came to the door, and came to actually kind of like it. Months into this arrangement, I visited a friend of another friend who had dogs that were not golden retrievers. Two pit bulls, which he cheerily informed me had recently killed a groundhog in the back yard. And I was able to handle that visit without getting particularly stressed!
I think my cynophobia wasn't quite severe enough to trap my prior beyond where innocuous experiences could shift it, and I had a strong exogenous factor — wanting to get out of an unlivable apartment and into a really cheap and nice room — that motivated me to adjust my views.
In the past few years I have intentionally cultivated more trapped priors. I was directly inspired to this by an old scott post: Epistemic Learned Helplessness or something to that effect. On certain subjects I know that a) I am not equipped to tell truth from falsehood in this domain; b) lots of people around me are actively trying to deceive me; and c) they benefit from my deception. Consequently, my defensive reaction to this is to categorically write off everything as fake in that domain, and fall back on my priors.
I have cultivated this intentionally as a response to a common failure mode of rationalists, where they are willing to extend charity far beyond where it's warranted, and bad-faith actors use this to manipulate. I am not getting manipulated anymore.
YMMV, just sharing my experience
> I don’t know why this doesn’t happen in real life, beyond a general sense that whatever weighting function we use isn’t perfectly Bayesian and doesn’t fit in the class I would call “reasonable”. I realize this is a weakness of this model and something that needs further study.
Since I hang out with engineers all day, I'm primed to think everything is feedback loops: Could we conceptualize the "trapped prior" as a problem in which an output signal (e.g. dogs are scary) is bleeding back into the input channel? The weighting function might be fine, but the problem is what counts as input? Would this help us identify a biological basis for the phenomenon?
The Lord Ross and Lepper study is great, thanks.
I'd been citing "They Saw a Game" for this, which I think makes elegant points about bias shaping perception. But at the end suddenly seems to conclude that we live in some kind of shared multiverse with mutually inconsistent realities just stacked on top of each other as the most likely coherent explanation for "sometimes people disagree about stuff." So it's nice to find one less committed to the "what even IS reality man?" trappings.
For "the rationalist project"... who is the primary target for this intervention on stuck priors?
Do you want the average ACX reader to be more aware of their own trapped priors and find tools to overcome them? Or do you want to find a better way to talk to passionate partisans, maybe create ways to disrupt common patterns and scripts, to hop the conversation onto new tracks that are more productive?
Just because I think the viable solutions will differ dramatically (unless you're really into spiking coffees), so that affects the search space.
In most arguments you're looking at both the content of the argument (data + logic) and the person/ context providing that argument. The updating process might get derailed when the subject doesn't believe or trust the argument source.
It prob doesn't apply to trauma, but for strongly held beliefs that seem unshakeable in econ, politics etc, does seem to play a large role. I don't know what's trapped here, whether it's the prior or perhaps it's the updater that also considers the validity of the source of any new argument. This applies when it's still a neutral source since that would *still* require you to at least believe your ability to update is not somehow screwed up.
OK, so the McGurk effect didn't work at all on me - I hear the same thing regardless. Anyone else having this experience?
It seems to me that the ad funded social validation slot machines that we call "social media" are trapped prior generating machines.
Another factor is that a lot of the time the evidence channel really is quite narrow.
I know someone who is convinced that vaccine side-effects are very dangerous and downplayed. I thought about trying to put together an actual analysis on the subject and presenting it with "I'm not asking you to trust 'them'; I'm asking you to trust me, because I actually checked" but then I realized I can't do it. I could probably get the post-approval surveillance data, but I don't have the skill to construct a demographically-matched control. Nor can I look at someone else's analysis and reliably tell that it was done correctly. Nor can I look at another analyst and reliably judge them regarding statistical skill and virtue of lightness.
Similarly, someone with a fear of dogs can find one easily, but what if you have a fear of sharks and the local aquarium doesn't have any?
So far so harmless, but how much US policy is written by people with a pathological fear of muslims who have never actually met one? Or who figure those chose to move to the United States are not representative of the ::handwaves at middle east::. That last point isn't actually wrong.
Is it possible to be so homophobic that even openly gay people refuse to out themselves to you? I bet it is.
I used GIMP to join the top half of the dark image with the bottom half of the light image, and the chess-piece pixels do in fact appear to be the same in the two images.
Nice article. I wonder if you could differentiate partisan results based on argument style — analytical arguments vs emotional appeals vs anecdotes. Logos vs Pathos vs Ethos.
Also, were there really accusations of dogwhistling on Biden's BLM support? I haven't seen that before, and a cursory search didn't find anything either.
A couple interesting/fun papers by philosophers on issues like this beyond perception, from a non-Bayesian perspective:
https://www.jstor.org/stable/20620131?s
https://www.cambridge.org/core/journals/episteme/article/abs/echo-chambers-and-epistemic-bubbles/5D4AC3A808C538E17C50A7C09EC706F0
(sorry for the paywalls; googling the titles will get you penultimate drafts)
Also check out Susanna Siegel's book Rationality of Perception
I'm not convinced that trapped priors explain the capital punishment experiment. An alternative model: people's certainty about the topic reflects how well they think they can justify their opinion.
If you have a strong opinion you want to be able to talk at length about why that opinion is correct. If you barely have anything to say you look foolish.
A study that confirms your opinion is something you can cite in support, that therefore lets you get away with a stronger opinion. A study that disconfirms it is not as helpful, but it doesn't make you less able to support your opinion. Possibly you can still wring something useful out of it, but at worst it doesn't change anything.
In this model the subjective probability isn't affected by evidence, just the social expression.
Your comment about gradually increasing low level exposure getting around the strong prior made me think of the descriptions of radicalization. Eg you start with a strong prior against holocaust denial, so would reject a direct denial argument, but if you are slowly introduced to related arguments then in time you'll be willing to accept it.
Which raises the larger question of if these techniques are content neutral. Using the same technique as reducing fear of dogs could one be trained not to be afraid of being hit by cars, so walk in front of traffic?
Your discussion of left-wing and right-wing priors makes it all the more clear why it is so difficult to be a libertarian. To us, almost everyone we deal with is ludicrous. It is very frustrating having a political conversation with 95% of the public.
Great article. It makes me think two things:
1. People should aim to have as few priors as they possibly can, and to chose them very carefully.
2. Priors can be dislodged in many ways, but the overarching theme is γνῶθι σεαυτόν (know thyself).
As I was reading the article my mind immediately went to PTSD before it was even mentioned. As someone who has had to do serious personal work because of PTSD, I was definitely dislodging a lot of trapped priors (about myself and the world), but I think one reason they were there is that they helped me keep the world much simpler. Dislodging a trapped prior (as someone who has had to do a lot of it) is really painful because it probably is naturally going to dislodge other beliefs and create a more complex picture of the world that will require a lot of work to ingest.
Re. the wine effect, I noticed many years ago that if I forgot what drink I'd poured into my cup, as I often do, that on drinking it I wouldn't say 'Oh, this is apple juice", but "Omigosh what is this strange taste that I've never tasted before?" Then, when I realized it was apple juice, the taste would suddenly shift to conform to apple-juice expectations.
I like to imagine that forgetting what's in my cup provides me with a "more-true" sense impression, but that desire is probably based on a bankrupt epistemology.
I think this is somewhat related. I went through a really really rough patch of trying to recall a traumatic event I had blocked out. I had told a loved one about it. A month or two later we tried talking about it again. We started off on normal conversation and then when I knew she was about to bring it up something happened. I felt like I lost my hearing. Like a mental discontinuity happened and no audio from her reached my ears. I only remember feeling like I was shriveling up on the inside. After her sentence I did what I could to quickly change the subject. This post reminded me of that. I feel like I have a better grasp on it now, but at that moment It literally seemed like I went deaf to stop hearing the words I wasn't ready to hear.
I wonder if there's any other cases of that happening out there.
It seams obvious to me (which means it is probably wrong) that the reason the prior doesn't update and the bandwidth on the sensation is throttled is because the person experiencing the situation has a flood of additional sensation that the context/prior weighting is generating.
The Rottweiler is scary. I am not only perceiving a dog wagging its tail, but also perceiving that my heart rate is accelerated, my muscles ache and want to pump, my hands are clammy, etc. If I can only pay attention to so many things at once, some of the immediacy of my physical reaction is going to take up my perception. Furthermore I'm now going to add an additional context on top of "dogs are scary" but also "when my heart races like this, it isn't a good thing" and "when I act scared my partner/parent is ashamed of me (which in shorthand is heard internally as: "I'm gathering shame right now")"
Exacerbating "bitch eating cracker syndrome", I think, is the escalation of moral investment in the designation "bitch." If you slap an ostensible bitch for eating a cracker, then you better be right that she's a bitch... otherwise you're a moral monster. In fact, you may need to up-level the designation to "evil bitch" in order to resolve your cognitive dissonance.
This is my hypothesis on how people turn abusive, or at least how my (ex) husband went down that path. In many respects, he is/was a very good man. I think he was trapped in the belief that I was an evil bitch, because what sort of monster physically abuses a non-evil human? This may be especially challenging for him because of his splitting - his world is full of angels and devils, with little in between - and thus he had a long way to fall. As trite as it sounds, I think the only hope for him is to learn to accept and forgive himself. But I'm not sure this is in the cards... may god have mercy on his soul.
Anyone that doesn't thoroughly hate both political parties is either a billionaire or not paying attention.
Possible editing error: In the section "Trapped prior: the more complicated emotional version", you introduce van der Bergh's idea that negative emotions reduce the bandwidth on sensory input. Then in the section "Reiterating the cognitive vs. emotional distinction", you introduce this same idea again, phrased as if it were the first introduction.
What is the usual time frame of studies like this on the entrenchedness of priors? Just typical minding based on myself, I can definitely see and feel my skepticism tingling whenever I am presented with evidence that I am wrong, but in the long run the political stances I tend toward have changed quite a bit, say over the past 20 years. I don't think I ever particularly strong stances and I still don't, but in general temperament they have changed. I'm not claiming to be abnormally rational or anything. I think I just don't care much and find myself expressing agreement with whatever happens to be considered acceptable to my peers, and those peers have changed over the years.
On the capital punishment thing, what was the evidence these people were presented with? This seems like a question of values. Values are fundamental, not based on evidence, so I don't know what should be able to change them. This feels like a manifestation of the is/ought gap more than trapped priors. For questions of value, all you have are priors. If they ever update, it's by fiat or epiphany, not evidence.
I'm reminded of KGB defector Yuri Bezmenov's take on trying to reason with demoralized people; "you cannot change their mind, even if you expose them to authentic information". Scott enumerates some potential ways to escape trapped priors. But if I were to channel Bezmenov's criticism of the Soviet Union I then have to ask, "is it possible to take advantage of this belief model to better convince people to defend your position"? I assume so.
Reinforcement of priors is one approach. For example, I really want this person to be afraid of dogs. Therefore, I'm going to keep showing them pictures of scary looking dogs. This is more or less "I'm going to read about how bad the other political side is from the people who agree with me already." (Is it?) But under this model anything that can decrease the strength of the raw sensation is another approach to convincing people. But this seems to be a more defensive strategy. However, I'm having trouble thinking of how someone would craft a situation where this is the method of attack. Thoughts?
Wouldn't it be great to just be the nice dog instead and only be conditioned on whether an action results in a treat? Sounds like a pretty nice prior if you ask me
Remember JJ Gibson, 1950's, mechanism of perception and organ participation, which shows that perception is a compilation of the person's environment and how the person interacts with it.
Observation on the extremes of the dog whistle problem. Policies have lots of consequences, both benefits and costs/positives and negatives, potentially even to the same people.
When a pattern of consequences arises which (at least seems) like it was not stated in the "This is what the Law is About" section (haha) - but the pattern is either really positive or really negative for some group or individual - the question is posed, Did they do it this way on purpose, by accident, or what combination? Did "they" not realize this would happen, or was it what they were after all along and they just hid it? Or was it not the primary goal, but a desired objective that the architects were aware of? Individual lives have that, someone can legitimately not realize, and at an individual level the person can also have motives they are unaware of - and they can also have definite motives and feign innocence.
The problem is, looking at the complex of unintended consequences of a policy created by a group, and then choosing the "every supporter meant this and knew it" option.
From the position of someone affected by the unintended consequences, if something is negative enough and goes on long enough, there is a "they reasonably should have known by now" position, which makes it possible to stop extending any benefit of the doubt to the architects of the policy. It allows someone to attribute intent and therefore be justified in loathing.
From an education point of view, sometimes people really don't know and don't realize.
I have to stop writing this. Maybe more later. Very interesting post.
So, I work as a Clinical Psychologist in the UK, and have been recently treating anxiety, including phobias and OCD using standard CBT approaches.
In the case of a phobia, the thing that limits people’s raw experience is typically avoidance of the phobia stimuli. This temporarily relieves anxiety, and thus strongly negatively reinforces the avoidant behaviour (by making the scary thing go away) and of course reduces the experiential bandwidth that the “dog” is not scary, meaning that they only have their prior to make predictions about future dogs with.
I’ve never used flooding, but my understanding is that it actually *can* work, so long as the person doesn’t freak out and avoid the phobia stimuli. As we have to do therapy ethically, if a person simply doesn’t want to be locked in the room with the Rottweiler, we’ve gotta let them out. But if we let them out at a moment of high fear, we actually increase the fear conditioning and avoidance. In order for a person to become habituated and de-conditioned to the phobic stimuli and past avoidance, they need to stay with the stimuli for long enough that the experiential data demonstrates it is not dangerous *and* for their amygdala to calm down. This typically takes 20-30 minutes, and I explain all of this to patients at the start. We don’t use flooding because few people could tolerate it enough to consent, whereas most people can tolerate the cute puppy stage of systematic desensitisation which enables them to gradually face more scary phobic stimuli.
The issue though with phobias is that the person understands that their phobia is in some way silly or harmful, and is prepared to do systematic desensitisation to overcome it. My patient who believed she had to check her children were breathing multiple times per night or they might die *knew* on some level that her belief was irrational, and wanted to challenge it, but the trapped prior and fear and alleviation when she checked kept her negatively reinforcing the checking behaviour. Exposure and response prevention changed that, but it was because she ultimately wanted to change and simply needed the method and a good therapist to guide her.
My sense is that some political beliefs could be similar. If you have limited data on a group of people and fear them, you will likely avoid them, both negatively reinforcing your avoidance and also not gathering data which could disconfirm priors. If you actually got to know some Republicans, or some Muslims, you might not find them to be that bad, if you actually spent long enough with them, but your avoidance could prevent you from ever doing this.
Our confirmation biases could positively reinforce seeking information that makes us feel better about reading information that agrees with our beliefs - we all want to feel we are correct. Conversely, the cognitive dissonance, or discomfort we feel when presented with information that demonstrates to us that we are in error could cause us to avoid such information, negatively reinforcing the avoidance and also limiting our exposure to such evidence. It could also provoke a defensive behaviour, such as stating that the evidence is in some way “lies” or “fake news” as to truly accept the data as being correct involves some discomfort or potentially embarrassment at realising one was wrong.
Potentially, the cognitive dissonance or embarrassment is even stronger the more outlandish and bizarre ones beliefs are, which could explain why people would rather continue to believe e.g. Qanon conspiracy theories than face the emotional discomfort for *long enough* that the discomfort fades, enabling them to continue to absorb new experience without avoidance.
I think the avoidance, time required for fear or discomfort to reduce, and defensive reactions could all be reasons why humans can’t simply absorb additional data and thus more rationally update. I hope this is a useful addition to your own model. I appreciate that you will know the basics of this in psychiatry, but assume that you’ve not actually spent the time in the room with the patient and the dog to find out what makes it so hard for her 🙂
Reminded me of this story from the New Yorker on the (de?)conversion of a Westboro Baptist Church member via Twitter.
I'm surprised Eliezer Yudkowsky's method of updating posterior probability is not mentioned as an excellent way of updating trapped priors.
On "When Prophecy Fails", I'd suggest seeing this later paper by Lorne Dawsone: https://www.gwern.net/docs/sociology/1999-dawson.pdf
In short, yes, people *can* become more fervent believers after a failed prophecy, but in general what actually happens varies. The paper looks at a number of groups that had failed prophecies and some continue and become more fervent, others fall apart, etc. I'd suggest reading the whole thing, but I think the summary version is that it seems to depend on how the group's leaders react in the immediate aftermath of the prophecy.
It's worthwhile to spell out why confirmation bias isn't actually rational: it's a failure to draw distinctions between the evidence itself, and your interpretation of it.
If you know you have a phobia of dogs, then, when you see a dog as scary, that shouldn't give you any new evidence that dogs are a threat. You knew in advance that would happen! It would happen no matter how harmless the dog actually was. The problem is that you ignore the layers of interpretation, and try to update as if you had encountered an objectively dangerous dog. If that had happened, it would indeed be evidence that many dogs are dangerous.
If you have a prior that vaccines are safe, and you hear about someone who died soon after being vaccinated, you will say that it was most likely a coincidence. This is a correct conclusion. Then, since this death was a coincidence, you will not update your priors et all. This part is wrong! The event is still a valid bit of evidence. It is something that would be more likely to happen if the vaccine were dangerous than if it were safe. You need to update (a tiny bit, in this case) in favor of "dangerous" - even though you are correctly confident that there was no connection.
Updating based on evidence, and the interpretation of said evidence, are two different processes that need to be kept completely separate. This is something humans are very bad at doing, hence "confirmation bias".
I'm not totally convinced by the move from subconscious trapped priors (snap judgments, sense data pattern recognition, phobias, PTSD, etc.) and conscious trapped priors (bias, beliefs, ideologies, etc.). The former is driven by raw associations and experience, but the latter arises from more sophisticated psychological constructs: identity, temperament, tribal affiliation. Rewiring the former seems doable and in some cases therapeutically useful, rewiring the latter seems both difficult and also perhaps not terribly desirable.
Is it truly wise in starting the post by talking about priors in the context of democrats and republicans? I understand that the distorted thinking people have around politics is a big problem, but that same distorted thinking also makes it even harder for you to reach anyone suffering from it (I.E. a large fraction of the population) when you start out the gate with "but what about reasonable republican arguments." They'd get angry and say that it's entirely justified for them to be aggressively opposed to Republican policies, and list any number of reasons why giving the benefit of the doubt to Republicans is like giving it to Nazis. I actively try to avoid news articles most of the time and I've read and enjoyed several hundreds of your posts, and it would be basically effortless for ME to make that list. It's an urge that has to be held back, an urge that as you say, seems reasonable to a person from the inside. And most people can't hold it back. So why start an important article like that, and lose a fraction of the audience from the start?
I mean, I understand the other failure mode. Give all uncontroversial examples and people will nod their heads along to your point and then as soon as something controversial comes up they won't apply the lesson to that one at all, act like it's a different case or just inapplicable. That's a tradeoff I suppose. But what does the side you chose on this particular tradeoff benefit you? it seems like democrats vs republicans was an example you picked because it was easy to reach for, not one chosen to be particularly illustrative. You don't do anything with that paragraph that could not have also been done by substituting in any other ideologies, and almost any other ideologies would be less rage invoking (less prior bending?). I don't feel like slipping the democrats vs republicans thing in at the start does anything meaningful to break political tribalism, but it will negatively tint the perspective of everyone who is not already extremely open minded.
(Oh, or maybe this is the kind of thing you're doing because your actual goal is to try to win over more republicans to your line of thinking, in which case perhaps it's a more useful paragraph than I'd thought)
I guess that element of the banner image is about updating priors instead of the Astral triad of Yesod, Hod, and Netzach after all. Alas, I suppose it was just a coincidence.
>I've sort of lazily written as if there's a "point of no return" - priors can update normally until they reach a certain strength, and after that they're trapped and can't update anymore. Probably this isn't true. Probably they just become trapped relative to the amount of evidence an ordinary person is likely to experience. Given immense, overwhelming evidence, the evidence could still drown out the prior and cause an update. But it would have to be really big.
>Sloman and Fernbach might be the political bias version of this phenomenon. They ask partisans their opinions on various issues, and as usual find strong partisan biases. Then they asked them to do various things. The only task that moderated partisan extremism was to give a precise mechanistic explanation of how their preferred policy should help - for example, describing in detail the mechanism by which sanctions on Iran would make its nuclear program go better or worse.
My take on this is that prompting someone for the mechanistic explanation confirms both that that 1) the belief directly serves its holder in terms of expected evidence and 2) there is a *specific* expectation whose negation might be evidence against the belief. When the context is generating predictions rather than commenting on someone else's work, it skews the utility away from belief-as-attire and removes the feedback loops of interpersonal politics that would drive people to extremes.
(Maybe there's also an instinctual element of leaving a line of retreat that become easier with moderated beliefs, but I'm not sure if that's strictly necessary for the effect. )
Unofficial book review survey: https://forms.gle/1aU6BdcAt5ZKit7n7
I'll send the results to Scott.
One issue is that your model may be too simple - in particular, there's more than two inputs.
Phobias are fairly simple - someone has an irrational fear of some stimulus. It makes no logical sense, they may even recognize that it makes no sense, but they still struggle with it.
But when we talk about, say, scientific knowledge, one issue is likely that people become falsely overconfident in their scientific beliefs because if they are often right about *other* scientific things, that makes it more likely that they're right about *this* thing. That is to say, they think that they are "smart science people", they are usually right about most scientific things, so when there's some scientific thing that's controversial, they side with whatever it is that their political beliefs tell them to, regardless of the actual evidence, but because of their science knowledge, they falsely end up believing that their beliefs are correct because they have other science knowledge that is correct, and that incorrectly gets carried over into the biased domain of belief.
This happens with a lot of experts in general, really; people assume they are good at X and therefore know better about Y, even though, in reality, Y is primarily a matter of knowledge and so being good at X isn't actually all that relevant.
This makes it inordinately difficult to change their scientific belief because their brain, rather than correctly applying it back as "you don't understand this thing in particular", ends up trying to apply it back against "you don't understand this thing IN GENERAL", at which point the counterevidence of "I know all this stuff!" comes back in and thus greatly devalues or even reverses the signal.
Someone who isn't so sure about science is going to have an easier time changing their opinion about some of these things if they aren't capable of cognitively isolating the "good at science" thing, becuase they don't think that they ARE good at science, and so their priors are more easily changed because they don't have the same resistance to changing their beliefs.
I think that people who can overcome this are the people whose view of science is "I am smart about science, so questioning this thing that I/people/the public/my tribe assumed to be true means that I am good at science". This causes the negative reinforcement loop to instead become a positive reinforcement loop, as it means that questioning common assumptions means you are *good at science*.
Of course, there is a downside to this, which is that if you combine this viewpoint with being not actually very good at science, you can end up believing stupid things or else having overly low confidence about everything, because you aren't very good at filtering out bad information.
Honestly, I sometimes wonder if this is why we have so many dumb people about some of these things - a lot of the antivaxxers I've met IRL are not vehmently anti-vax, they just are distrustful of them because they know enough to be suspicious of medical studies but not enough to actually be able to apply skepticism correctly to data. They're skeptical of everything, and would, ironically, be better off with a worse basic herustic, because they aren't actually good enough at "all things should be questioned" to actually apply it usefully.
The people who trust respected authority figures like Dr. Fauci are better off than they are.
Meanwhile, the people who are *actually* good at this stuff were saying to wear masks before the CDC said to wear masks. My mom bought respirators back in January 2020.
This may be why these sorts of things are actually so common - because having fixed priors is bad, but being incapable of actually learning anything is worse. Someone with relaxed priors who is actually good at this stuff will probably only end up with a few weird beliefs, if any, but most people are better off with the worse heuristic because they're not capable of using the better one.
This would also suggest that using psychedelics is probably a bad idea - permanently trapping people's brains open is going to probably lead to net negative consequences in most cases, as while there is a greater optimum to "open minded + competent" than "closed minded + competent", "open minded + incompetent" is probably worse than either.
Which is probably why most people are closed minded rather than open-minded, but there's enough people who benefit from being open minded that the trait hasn't been removed from the population.
Thank you so much for laying this out.
I have pondered this phenomenon as a layperson for quite some time; my intuitive thought was that at some point the Bayesian prior is so strongly believed that contrary evidence is taken as an indication that the source of the contrary evidence is unreliable, not that the prior itself might be wrong. I'm not good enough at the math to model that, but any given observation in the Bayesian model doesn't have to be viewed as either true or false; it (like anything else) is assigned a likelihood. But, intuitively, if the NY Times reported (seriously) that the earth was flat, I would not decide the earth was flat, I'd start doubting the NY Times (more than I do; lolz).
FWIW my internal shorthand for this phenomenon is not "trapped priors," which is fine, but rather getting stuck in a "Bayesian Black Hole." If you like the "Bayesian Black Hole" framing, feel free to use it - no attribution needed!
Thanks again.
Is it the case that when we're young, our minds run sensation and context through a sensation-weighted algorithm; and once we're older the algorithm switches and weighs context more heavily? Is the context vs sensation-weighted algo thing the same as low vs high neuroplasticity?
You observe that, even with a very strong prior, a "reasonable" update function should be able to notice that the experience was at least marginally less bad than predicted, and slowly update away from the prior. But that empirically, this often doesn't seem to happen.
I feel I can very easily imagine a minor and subtle defect in the update function that would cause this failure. For example, suppose evolution "wants" you to update 20% more rapidly on negative stimuli (for similar reasons to why we have loss aversion). The way that *ought* to be implemented is: calculate how far your beliefs would move "normally", then multiply the movement by 1.2 (this will always cause you to move in the same direction as before, just farther). But suppose the *actual* implementation is: multiply the "badness" of the perception by 1.2, and then update on that. If your prior was approximately neutral, this works out to nearly the same thing. If your prior was bad, but the "perception" (weighted combination of sensation and context) was good, then the multiplier isn't used and you update normally. BUT if your prior was bad, AND the perception was only *slightly* less bad, then multiplying it by 1.2 will make it *worse* than your prior, and cause you to update in the wrong *direction*.
That second implementation isn't *defensible*--you couldn't rationally argue it is somehow preferable to the first implementation. But I can easily imagine a lazy programmer doing it without realizing the issue, checking that it gives expected results in some basic tests, and then calling it done.
Evolution is certainly not MORE foresighted than a lazy programmer.
But even that explanation might be overcomplicated. When I read your description of the predictive coding theory of the brain (probably this was in your review of Surfing Uncertainty, though I'm not certain), one detail that REALLY REALLY jumped out at me was when you said the brain compares the sensory input (with error bars) to its prediction (with error bars), and if they match, then it sends THE PREDICTION (not the sensory input!) up to the next-higher layer of abstraction for further processing.
That was just your summary; I don't know how much the evidence supports that detail, or even if you were consciously making the distinction when you wrote it. But if that gray box in your diagrams isn't so much taking a weighted average of the inputs as using some threshold function to decide which input to IGNORE, then that would compactly explain the failure to update in the direction of the sense input.
(Disclosure: I immediately thought that this detail would make sense of a lot of arguments over board game rules where both sides are convinced the rulebook supports their own position, so maybe I was predisposed to believe it.)
The shortcoming of such an approach is that so many - and arguably the worst - biases are not based on direct experiential "priors" at all, but on things like values and sensibilities. So even if one found a way to help people prioritize new "raw" experience, would it even touch the problem? This of course particularly applies to politics.
Having said that, the research on psychedelics does suggest the more "raw-like" experiences it facilitates leads to a coalescence of a sort around more relational values. Though the quality of these studies hasn't been that great, and likely many went into the experience with a particular openness to being changed.
An openness that many if not most of those with "trapped priors" or rigid values are unlikely to share.
If one is aware of trapped priors and how they functions (such as you've described), can that help make us less susceptible to them? Not necessarily for traumatic things, like the "scary dog", but the more mundane, like political bias and polar bears?
A lot of my strongly partisan/conspiracy theory-ish beliefs are things I just kind of choose to believe because it feels good to believe them. But I'm capable of setting them aside in favor of more reasonable beliefs when I need to. It's not really trapped priors or any kind of issue with rational thinking. I'm just choosing in these moments not to think rationally.
"(My usual metaphor is "if God came down from the heavens and told you ... - but God coming down from the heavens and telling you anything probably makes apocalypse cultism more probable, not less.)"
What if God came down from the heavens and told you He doesn't exist?
Interestingly, you see a similar sort of sensitization and desensitization with allergic reactions. After an initial sensitization to certain allergens (from e.g. insect stings), subsequent allergic reactions can be significantly worse. Despite that, allergy shots can also work for those same allergens, with the main difference being the amount of the allergen you're exposed to at each time - gradually increasing as you become desensitized to it.
It makes me wonder if the underlying mechanism is similar in each case.
The supplementary materials for the Sloman and Fernbach paper include the prompt they used to ask for mechanistic explanations:
“Now, we'd like to probe your knowledge in a little more detail on two of the political issues. This is the first one. Please describe all the details you know about [the impact of instituting a 'cap and trade' system for carbon emissions], going from the first step to the last, and providing the causal connection between the steps. That is, your explanation should state precisely how each step causes the next step in one continuous chain from start to finish. In other words, try to tell as complete a story as you can, with no gaps. Please take your time, as we expect your best explanation.”
https://journals.sagepub.com/doi/suppl/10.1177/0956797612464058/suppl_file/DS_10.1177_0956797612464058.pdf
So I may have already posted a comment here, but it's not appearing.
This is brilliant.
My intuitive sense is that we assign probabilities not just to priors, but to observations, including the likelihood of the source. I generally believe the NY Times, but if they reported that the earth was flat, I'd lower my estimation of them, not decide that the earth wasn't round. In this way a sufficiently strong prior can overwhelm contrary evidence by degrading belief in the observation.
I call this a "Bayesian Black Hole." If you like the term, feel free to use it.
Formal debate training, where one must research a proposition and then be prepared to argue persuasively either side, was once a common requirement in education. It helped foster critical thinking. Similarly, the Catholic Church adopted the concept of a “Devil’s Advocate” to argue against the elevation of a person to Sainthood. I have found, and there is much written about the technique, that formally designating a “Devil’s Advocate” helps break down biases in decision making. It helps eliminate the “you can’t talk about that” factor.
Van der Bergh et al see what they are studying as "Better Safe Than Sorry" information-processing strategies. Carrying that over to trapped priors the question is why it would be unsafe to release the prior and allow it to be updated. What safety is preserved by keeping that prior trapped?
As a coach, I have consistent success asking people what's at risk if they were to reconsider certain deeply held beliefs, and asking if there are better ways to mitigate or tolerate that risk than the strategies they've been pursuing. (Scott's Mental Mountains post nods at similar interventions.)
This only works if the client is suffering in some way because of their held prior, and has reason to question it. But it could work for rationalists as well: asking "What's at risk for me if I were to change this belief?" may draw our attention to where our own reasoning might be motivated.
This is a great write up, but I think it overlooks the significance of the degree to which this phenomenon exists among the public. It's easy to see how this can occur in political partisans and people with phobias, but what is rarely discussed is how powerful the phenomenon is on culture war topics, even with otherwise extremely rational people. For example: Hacker News and /r/SlateStarCodex are two communities with very high concentrations of rational people. However, if certain topics come up, individuals in these communities almost without exception are rarely able to keep it together and engage in a logically and epistemically sound conversation, free of rhetoric. It seems that people are simply unable to practice the Socratic method on certain topics.
I believe here lies an excellent topic for some research. Typically, studies tend to focus on average person, who usually know they are involved in a study. I think it would be interesting if a way could be found to seed deliberately controversial articles directly into such communities, and then have researchers observe how these rational people behave irrationally (including things like mind reading and predicting the future), and perhaps even intervening to see if any way(s) can be found to break them out of the illusion. And even if that process was a failure, then see what the reaction is when they are let in on the fact that they were the subjects of a study, and shown a copy of their comments, with logical criticisms noted along side each comment. If this could be done by someone who the community members respect, I think it could be very informative, and possibly lead to some form of a saleable approach.
"In the old days, psychologists would treat phobia by flooding patients with the phobic object. Got cynophobia? We'll stick you in a room with a giant Rottweiler, lock the door, and by the time you come out maybe you won't be afraid of dogs anymore. Sound barbaric? Maybe so, but more important it didn't really work. You could spend all day in the room with the Rottweiler, the Rottweiler could fall asleep or lick your face or do something else that should have been sufficient to convince you it wasn't scary, and by the time you got out you'd be even more afraid of dogs than when you went in."
This sounds odd; why would they apply a treatment that didn't work? Furthermore, when I looked it up, a lot of pages claimed that the treatment did work, with some even claiming that it was highly effective.
If you're interested in how doomsday cults respond to the lack of doomsday, Stephen O'Leary (at USC until he died last year) had research and a few books about that. He had put a group together to watch the world not end in the year 2000. I remember reading some of his [i]Arguing the Apocalypse: A Theory of Millennial Rhetoric[/i] at the time, but I never followed up after the world failed to end.
There's some evidence that, when someone has a relatively strong position in favor of X, ridiculous over-the-top propaganda from pro-X extremists can be more effective in making the pro-X belief less strong, compared to a reasonable argument or well-designed study refuting X. I remember reading about a study of attitudes about Israel/Palestine that came to that conclusion: people who supported Israel were made more extreme in their position after viewing a reasoned argument in favor of an independent Palestinian state, but they became more moderate after viewing over-the-top crush-the-Palestinians propaganda. Unfortunately I can't find the study now.
I know that for me personally, there are many cases where extremism from my own political camp was more effective in moderating my political beliefs, compared to arguments from the "other side."
Maybe this works because, since the content at first glance appears to confirm your beliefs, you don't reflexively decrease the weighting on the sensation compared to priors? I would suspect that this only works if you believe the extreme ridiculous pro-X propaganda to be genuine; if you think it's actually a straw man from the anti-X side, your pro-X prior will only be reinforced.
Scott, I have some articles on priors that are relevant to this post, but cannot find the email to reach you at. Please let me know which is best so I could send them to you!
I think part of the story that might be missing is that your priors can feed back and influence what sensations you even collect. Consider the fisherman who never casts a line into a certain part of the river because he "knows" that the fish never bite there and it would just be a waste of time. I wonder if human attention is gappy enough that positive interactions with a dog are literally invisible to the cynophobe.
What's going on with psychedelics in regard to priors seems pretty straightforward: they potentiate Bayesian updating.
It just turns out sometimes that's truth-directed, and sometimes it's not. This inconsistency of effect is not because of anything about psychedelics. Rather, it's simply because Bayesian updating isn't an oracle for truth.
Rather than seeking a "pro-rationality" intervention, one should seek practices of good mental hygiene, including some of those Scott mentions. The effective model is more like pursuing good diet and exercise, and less like debugging code.
This is a similar situation for NDE’s. People die and ‘come back’ different. Why? Their dying causes a release of trapped biases as to who they are supposed to be. A break in the narrative. Death exposes the absurdity of our notions about life and makes obsolete the fears that previously had such a hold over us, because we have now experienced the ultimate ‘happening’ of dying that what we previously feared seem absurd.
This is one of many reasons why we should be using the word ‘Steelman’ as much as possible. When it comes to trapped priors in the context of personal beliefs, political or otherwise, having to steelman the argument of the person you disagree with will probably help in reducing the strength of your priors (assuming they are actually capable of granting charity to the opposition). I’m so sick of people assuming the worst of members of the outgroup.
This thing—The Trapped Prior—is exactly the concept I've been looking for. This brings together a chunk of old rationalist material, and arguably The Cowpox of Doubt (https://slatestarcodex.com/2014/04/15/the-cowpox-of-doubt/) discussed the negative version of trapped priors, i.e. psychological inoculation. After hearing a large number of poor arguments for P, you will evaluate even objectively strong arguments for P is poor. This is it: The Trapped Prior has afflicted you. Your own thinking cannot recognize it; you will feel like "I evaluated this argument fairly, as I evaluate all arguments, and it's clearly a bad argument."
Another interesting line of thought is whether or not the Ethnic Tension-style arguments (https://slatestarcodex.com/2014/11/04/ethnic-tension-and-meaningless-arguments/) somehow act on people's ability to evaluate policy proposals by creating a context such that policy proposals for a certain position will get evaluated more positively.
SJWs begin the trapped prior process for people until they can no longer perceive wolves.
Were you surprised on 1/6? Perhaps it's because of your trapped priors around leftism.
Another possible method to explore is to pretend to believe in an opposing prior. Like a democrat could pretend to be a republican, or a cynophobic person could pretend to love dogs. I imagine a playful, humorous setting where you start out with a ludicrous parody of the position opposed to the one you're stuck in and end it with laughing about it. Then maybe extend it for longer periods of time or try to make it more "realistic". And only after you have some familiarity with the role, bring in the experience and try to look at it the way this role you're playing would.
Has that kind of thing been done in psychiatry?
You're assuming exactly two inputs per evaluation to the update function, and also that the function is itself immutable. Do you trust those assumptions? Should you?
On the subject of overcoming phobias there was a UK TV hypnotist/NLP practitioner called Paul McKenna who claimed to be able to cure people's phobias within half an hour or so. You can look him up on YouTube. He is not the most charismatic person but I do remember watching a person who went literally white in the face seeing a dog a hundred yards away and within the next hour they had been transformed into someone happy to have a dog leap up and lick their face. It was quite remarkable and I don't think it was a fake.
Scott said - ....self-serving bias. People are more likely to believe ideas that would benefit them if true ....... I suspect that these are honest beliefs
Elephant in the brain (Simler/ Hanson) explanation. It is an Darwinian advantage to sincerely believe one's own lies, so we've evolved to be really good at it.
On the therapeutic side, it seems like being in a room with a rottweiler while having cynophobia might be unpleasant enough that "dogs cause unpleasantness" gets updated by the experience... but naive mechanistic reasoning suggests that ameliorating contextual factors (like body armor, or the dog being chained up, and intervening protective glass, or also eating ice cream, or all of these factors at the same time) could modulate the process.
This feels consistent with a strongly felt intuition of mine that a FUNDAMENTAL pre-condition for learning, and thinking creatively, in general, is a (basically justified) feeling of "safety".
The most effective protection I can personally imagine is simply "a adequately powerful agent close enough to ensure my safety". Also, I think it might be common for people to have phobias for things a parent encouraged them to be afraid of as children (roughly) because the parent thought the fear would be more helpful than harmful?
When I google the plural with ["trapped priorS" attachment style] I find nothing. Modifying to search for the singular form gives a tiny number of ecology papers about wild animals being "trapped" "prior" to some measurement or treatment that the research paper is focusing on. I'm wondering if "trapped priors" is your own name for this? Are there other keywords for this that might turn up other veins of related research?
Related to parental judgement that a phobia might be good to install... I've talked with people who were phobic about various critters, and they basically have never been interested in curing their phobia. It is like they "want to find something bad tasting" because then they'll eat less of it, which would be good if it is objectively bad to eat? Except they are reasoning this way about fear and avoidance instead of taste and food.
This helps me make sense of why "talking through the mechanistic steps" around a big vague fear might cause people to become more mentally flexible: maybe visualizing more intervening details allows them to imagine a different action than "shunning the first step" which could ALSO protect them from the very last domino falling over in a way that they think will cause objective harm?
If they thought the fear was their only good strategy for avoiding harm, they might cling to the fear.
I think self serving bias along with the rich & powerful having leverage to change the political / economic system is enough to explain cost disease.
The longer everything exists in some kind of stable state the more people with power figure out ways to corrupt the markets & bureaucracy to their benefit. Citizens United, Surveillance Ad Tech, Corrupt Unions, CEOs making 300% more than workers. There won't be a single explanation because it's simply processes being corrupted by people with power who likely are deceiving themselves that this is a good thing.
"It can happen (in theory) even in someone who doesn't feel emotions at all. If you gather sufficient evidence that there are no polar bears near you, and your algorithm for combining prior with new experience is just a little off, then you can end up rejecting all apparent evidence of polar bears as fake, and trapping your anti-polar-bear prior."
I don't see how this can be true. If you're very sure there aren't polar bears, and then you keep seeing bits of white fur and tracks that look like they could be from a big bear, you might not notice them at all, or you might be confident they're fake, or that the fur is from a dog. But none of that can make you MORE confident that there are no polar bears.
The only way evidence of polar bears could make you more certain there are no polar bears is if it convinces you that, say, someone is out to get you by faking signs of polar bears to make you look crazy. In other words, an emotional situation.
Comparing phobias and political/cognitive biases is an interesting exercise. People with phobias may recognize they have an abnormal condition and can seek or accept help. People with unassailable political convictions often don't recognize their state as a problem, even when their beliefs are relatively fringe and impose real costs like the lost of family and friend relationships (QAnon, cults).
The difference is obviously in the nature of sensory evidence: after all, one can see dogs being walked everywhere by unafraid people, but it's impossible for a bystander to directly experience an unjust death penalty. Generally, the only way to absolutely know a person is innocent is to be that person.
I'd theorize that in cases of abstract/intellectual issues like politics the "raw experience" channel is fed by the imagination, not the senses, and the imagination is essentially just a remix of context priors, plus whatever one is trying to imagine. So context in => context out and beliefs won't move away from context
Then why would argument ever work on anyone? Presumably when one can imagine their conversation partner's specific argument more easily, allowing a path for non-prior concepts to enter the process. This naturally works better when the incoming concepts more closely agree with one's priors, or possibly for individuals that naturally give their priors low weights.
I'm particularly tempted to think that a link between low prior weights and a flexible imagination might also correlate with some people's ability to think much more abstractly than others.
You're conflating two things with very different mechanisms of formation because they both present similarly as "trapped priors". Cynophobia (or many traumas really) can result from a single bad experience and is generally more context specific. It's the result of one-shot learning. A rat learns the association between the bell and the shock in as little as one trial: the limbic system at work.
Political biases are the result of much more evidence accumulated over a much longer period of time, are far more context-independent, and are the trigger *to* intense emotion rather than being triggered *by* intense emotion. A much better model would be that of habit formation. The mechanisms of learning/unlearning are far less plastic in this case, responses are far more reflexive and the emotional component is secondary.
Just for fun I made a quick spreadsheet to simulate what would happen if your priors updated slowly.
Say you're a perfect Bayesian agent embodying Bayes' theorem. You start out very certain that all dogs are scary, and it's very unlikely that happy puppies even exist, but then you see an obviously very nice puppy. You'll update immediately from 99.9...% confidence that dogs are scary, to something like 1% odds that all dogs are scary.
But if you modify the update such that the experience of seeing a happy puppy only updates your prior half as much as much as it should, it takes you ~4x as many exposures to the happy puppy before you reach the "ideal Bayesian agent" confidence that happy puppies exist.
And if your prior only moves 10% as much as it should, it takes you *~25x* as many exposures.
Given my old days in a neuroscience lab, I would conjecture that a trapped prior can be facilitated by long-term potentiation and long-term depression (https://en.wikipedia.org/wiki/Long-term_potentiation) in whatever synaptic connections that are involved, creating a lasting link that is resistant to changes in stimuli. If so, then a more promising way to change the behavior would be the pharmaceutical or therapeutic interventions that reverse LTP/LTD in specific brain regions, like maybe an NMDA receptor blocker (memantine?), and whatever EMDR does, possibly. And then we would finally get a handle on the addiction to unhealthy Vietnamese noodles, and other pho-bias.
The “if God came down from the heavens and told you” scenario plays out amusingly in “The Oven of Akhnai”, a story in the Talmud.
To paraphrase, two rabbis are arguing over whether a new type of oven is kosher. Eventually Rabbi Eliezer gets frustrated that his arguments are failing to persuade Rabbi Joshua, and he declares, “Look, if my interpretation of the Law is correct, this carob tree will move.” At once the tree leaps a couple of yards to the left and re-roots itself. But Rabbi Joshua just replies to the effect that strange trees hopping about is no basis for a system of determining ritual purity.
So Eliezer says, “Well then, if I’m right, the river will reverse it’s course!” And the river promptly turns and flows toward its source. But Joshua remains unimpressed.
“OK,” says Eliezer, “If I’m right, the walls of the yeshiva will tumble to the ground!” And they do start to fall - until Joshua scolds them for getting involved in a halakhic dispute without proper training. (The walls then hedge their bets by standing up again, but not quite perpendicular.)
Finally, Eliezer has had enough. “If I am right,” he cries, “the very Heavens will declare it!” And from the sky an immense voice booms out, “ELIEZER’S RIGHT, ALRIGHT? JUST GIVE IT A REST ALREADY.”
But Joshua replies simply, “Torah is not in Heaven.”
You’d think that this extraordinary assertion of the primacy of evidence over the highest conceivable authority might have licensed an early flowering of scientific reasoning in Jewish culture... but sadly the idea doesn’t seem to have been extended much beyond, “We should bicker incessantly about liturgical texts.”
https://en.wikipedia.org/wiki/The_Oven_of_Akhnai?wprov=sfti1
Simulated annealing is a simulation of real annealing, which is a metallurgical technique. After beating a piece of metal for a while, some of the atoms in the original piece get dislocated. This means atoms actually get pushed around and crammed into open spaces in the crystal lattice. This makes the piece slightly denser and also more brittle.
To overcome the brittleness, the piece is briefly reheated. This causes some of the connections around the dislocations to break and new ones to form. This makes the piece less brittle while also preserving the atoms-forced-into-nooks-and-crannies density that came from working the piece. The final piece is denser than the original yet not as brittle as it was before annealing.
Simulated annealing is a technique in computing which is designed to escape local extremes an optimization algorithm might get trapped in. It takes lots of forms, but a common one is to speculatively "teleport" your current position in the space you are optimizing to another semi-random location. This briefly makes the optimization worse, in the hopes that further optimization from this new location will yield a better global result than the previous optimum. This is analogous to reheating the metal and allowing it to recool, where the temperature of the piece functions the same as the cost of the optimizing cost function.
Annealing is both contexts is a process of making a crystallized structure more plastic in the hopes that it recrystallizes into a better form than the initial.
Here in Amsterdam University there's this professor Merel Kindt (https://www.uva.nl/en/profile/k/i/m.kindt/m.kindt.html?cb) who's doing the not-gradual-barbaric-exposure followed closely by propranolol and says that works wonders for both phobias and PTSD. Here is her clinic's website: https://kindtclinics.com/en/
I'm an engineer and not a doctor/psychologist/etc so wouldn't know a thing, but the theory kind of makes sense and your post made me remember that, the underlying mechanics seem similar.
I tried to convince my girlfriend to go there for her extreme dog phobia, but even the thought of the therapy triggers her trapped priors so we couldn't try that yet. That's kind of a recursive trapception we need to take ourselves out from. One last detail is, when she's really drunk she loses the phobia (and pets dogs), but it comes back when she's back sober.
Here is an American newspaper link to the clinic:
https://www.washingtonpost.com/news/morning-mix/wp/2015/12/15/scientists-say-theyve-found-way-to-cure-fear-of-spiders-in-2-minutes-could-they-also-cure-ptsd/
Seems to me you are kinda confusing two different models. On the emotional model we are imagining an agent who engages in some kind of reward based learning. If you allow a feedback from the model to the learning function you can easily see how it would end up in an state where objectively ok experiences were actually interpreted negatively.
On the Bayesian rational updating model there shouldn't be any means for a high prior to trap your beliefs. And that's not just an assumption of perfectness but seems like you'd have to intrinsically build in this kind of bad behavior.
The problem is simply that the more certain you are that *all* dogs are horrible and dangerous the higher your prior should be that the dog will attack you and thus the greater the evidence updating will give when the dog doesn't bite. You can't explain why the individual would keep predicting a high chance of dog attacks even though they don't happen.
I suspect, that if you actually ask people it's not that they will report crazy high priors for any concrete idea. Thus, I think the take away should be that this case is better explained by the emotional reward idea first pretty well and the pure Bayesian prediction model (unless you are predicting emotional reaction in which case you just grated on the reward model) not so well.
This semi-relates.... I have said to people in the past that if their priors are either 0 to 1, they never get updated.
I don't think psychedelics are necessary to assist in changing priors more rapidly, but really any mind-altering substance that induces a pleasant emotional affect. MDMA would likely work much faster and more effectively than psychedelics. Putting someone who is afraid of a dog in a room with a dog is just as likely to terrify them as cure them, but put them in a room with a dog on MDMA and they would probably not be able to resist cuddling or at least would have profoundly reduced fear.
But even MDMA isn't necessary, alcohol works too. Alcohol works quite well for overcoming all kinds of mild phobias and social anxieties and opening one's mind to things one is generally against.
Another things that works pretty well is being under the influence of whatever neurochemicals are coursing through one's brain when they're in love. I've persuaded several exes out of life-long strongly held religious and political positions while they were in love with me. Falling in love or maybe even lust or strong like with someone who holds convictions you disagree with is a strong motivator to becoming open to changing them.
In general though, your post (which I liked a lot) was somewhat depressing, as someone who likes to take my dogs hiking, I am always trying to find a resolution for people who are irrationally afraid of dogs. Because I don't like being screamed at by someone who is terrified, as has happened. One of my dogs looks like a big black wolf, though he has the most bite-inhibition I've ever seen on a dog...even as a puppy he refused to put his teeth on skin...and poses essentially zero risk of ever biting a human, but he looks scary to people afraid of dogs.
I have always wondered how it is that so many people who fear dogs claim to have been "attacked" multiple times, which is statistically extremely unlikely for anyone who doesn't make a career of working with them, and I've suspected that what they consider an "attack" is exaggerated -- i.e. a dog running towards them or barking or making noise. Your post indicates that is likely the case.
Though it is even worse than just a matter of trapped priors, however, because people with this phobia often behave in ways that make their priors much more likely to be confirmed. People afraid of dogs behave in a manner that dogs interpret as "sketchy person who is acting weird and is a threat" and dogs then become more likely to actually growl at or menace the person. This certainly is the case in human social contexts. If someone has a prior that "X category of people discriminate and look down on people like me", they are likely to behave defensively and in an unfriendly manner -- i.e. not making eye contact, scowling, using a hostile tone -- which then increases the likelihood that they get a like response, which then confirms their prior. Hence the chip on one's shoulder ends up being realized.
What a terrible cycle to get out of between competing political groups. One's priors leads one to assume the other person is going to lie or be an idiot, and so do theirs, you both interpret whatever the other says through that filter, AND you both actually behave in a ruder and more hostile way than you normally would, thus lending both assumed and actual validation to the other's priors. What a shame that nowadays, it is fashionable to preface one's argument with "speaking as an XYZ identifying person, I think that blah blah blah", which just sets up everyone's brain to trigger off the identification rather than the argument.
Great article, I’d suggest an alternative mental model that solves this problem:
> In most reasonable weighting functions, even a strong prior on scary dogs plus any evidence of a friendly dog should be able to make the perception slightly less scary than the prior, and iterated over a long enough chain this should update the prior towards dog friendliness. I don’t know why this doesn’t happen in real life, beyond a general sense that whatever weighting function we use isn’t perfectly Bayesian and doesn’t fit in the class I would call “reasonable”. I realize this is a weakness of this model and something that needs further study.
In my model, the brain’s sensory processing unit is a prediction machine. It receives raw experience, guesses how the higher units of your brain will feel about this input, and then processes the input to highlight relevant details based on its guess. If it’s right about what it guessed, the processing is reinforced. If its wrong, then it adjusts in the appropriate direction.
When you are trapped in the room with the Rottweiller, it guesses you will be terrified, and will want to highlight the parts of the experience that are potentially dangerous. Since you *are* indeed terrified, it guessed correctly, and the processing is reinforced.
However, when you are gradually introduced to puppies, your processing guesses wrong. The higher units of your brain are not terrified, and so the signal is sent down to the sensory processing unit to change its predictions for next time.
Direct persuasion is hard, almost downright impossible for strongly-held beliefs. That's because humans are social creatures, and our lizard brains are very attuned to the social hierarchies in our world around us.
If my friend Bob tries to convince me of something in direct contrast to my political beliefs, yes, maybe my political priors should be updated based on the strength of the argument and based on my perceptions of Bob's trustworthiness and intelligence, but at the same time my priors about Bob are being updated, too. And because we evolved as social animals, we are much readier to shift our perceptions of people.
And there's the fact that if I do change my belief, that also sends a social signal back to Bob, and he's going to update his priors about me.
The medium is the message, and when that medium is another person, interpersonal dynamics take over.
If somebody came along and advertised a "prior-unsticking therapy for improved rationality," the first thing I'd want to know isn't how they unstick my priors, but how they determine what I'm being irrational about.
And also maybe the opinions people tend to hold after embarking on this therapy, so that I can judge whether I want to hold those opinions myself.
Right now, we have deemed some opinions to be ethically medicalizable: personal delusions ("I am Napoleon!"), paranoia ("everyone's out to get me"), and others. Insofar as prior-unsticking is a therapy for such conditions, it seems unproblematic.
But if we're considering roping in opinions that are wacky, but not currently medicalizable, then I'm concerned. Even if everybody has a tendency to be irrational about politics, that doesn't mean that everybody (or anybody) should be subjected to prior-unsticking therapy over their politics.
Political irrationality eats everything, and it will eat this too. Just wait for the day when a decrease in support for an unpopular opinion is deemed as evidence for the efficacy of prior-unsticking therapy...
> I think van der Bergh argues that when something is so scary or hated that it's aversive to have to perceive it directly, your mind decreases bandwidth on the raw experience channel relative to the prior channel so that you avoid the negative stimulus.
How can this be true when e.g. meditation techniques tell you to focus on the actual sensations of the pain, which I find are less aversive once I force my experience channel wider? Pain goes from "ahh the pain" to "a sort of weaving blooming sensation in my abdomen, with unpleasant valence". It seems like explanations of the form 'helps you avoid negative stimulus' can't be right due to how often context makes things worse. I would faintly bet on it making thing worse on net – it feels like there's no experience so good that it can't be ruined by multiple bad experiences (ordering a dish I strongly expect to be good each time, that is slightly less good than it was last time, until I get sick of it), but some things feel so fixedly bad would take enormous effort to make them less aversive or scary.
Probably the 'ultimate reason' trapped priors happen has to be something about saving on computational costs, but IF it were adaptive/functional my guess would be that it's good not to update on certain phenomena – if you saw a tiger chilling out 100 times in a row and decided that tigers were mostly nonscary, you'd sort of be more correct than the tigerphobic villager but slightly less likely to survive.
But this seems wrong, because many things I get trapped priors about are totally harmless. Regarding BEC syndrome – weirdly, I've rarely had this for someone treating me badly. When someone is very mean to me it's "so extraordinary it needs conscious explaining", and I come up with best-fit explanations for their behavior. I do overall expect them to treat me badly again, but they feel like a complex system that requires complex thought in response. But when someone is MINORLY ANNOYING in the first few times I interact with them I can very trapped.
Could this also be an additional mechanism of depression, when it happens in fuller generality on *all* possible experiences?
Thanks for breaking down a really important cognition "bug" so clearly. I want to suggest a small modification to your framing, which I think would add more clarity.
Your current theory treats agents as performing Bayesian inference on a stream of experience. If you instead assume the agents are performing active inference, the theory becomes more concise.
In your model, the agent receives a raw stream of experience that it must integrate with its priors. Active inference suggests that the agent queries the world, using its priors to generate hypothesis. The resulting experience is then integrated with the priors via a Bayesian update.
So a regular person locked in a room with a dog might have a low prior on the dog being dangerous. They are actively scanning the room for all sorts of cues including: is this dog dangerous, does this dog want pets, is this the cutest doggie they've ever seen.
A person with cynophobia is also actively scanning the room, but they are already certain that the dog is dangerous, so instead they are looking for exits and trying to figure out exactly when and how the dog will attack them. It's not that this person fails to do a proper update on the experience, it's that their experience is actually very different.
The queries that the agent performs are also not purely epistemic. They are not just trying to have more accurate priors, but to also act in accordance with their values. This captures both self-serving bias as well as emotions reducing the bandwidth of experience. In both cases, the agent isn't trying to learn as much as it can, but is instead trying to be safe, avoid pain, and maintain its social status.
I hope this made sense, and I want to credit Belobog with posting this first with his fisherman example.
I am also not sure that a perfect Bayesian learner would slide into confirmation bias, and my intuition says this is not the case. Unless every dog experience was *as terrible as you imagined*, your prior would gradually adjust down. Of course, humans are not perfect Bayesians. It's possible that humans would score "there was a dog and I was really scared" and "there was a dog and I was really scared and it attacked me" the same, or that they can't distinguish between a prior of 0.99 and 1, and so the update is always too small to move the needle. But I don't think we need to blame Bayesian updating when active inference is a much cleaner explanation.
As a spacecraft engineer who specializes in Kalman Filtering, which is basically recursive Bayesian estimation, I have always enjoyed the parallels with cognitive processing. This article reflects a problem we often see in KFs when the filter parameters are not properly designed, and the filter insufficiently weights new sensor inputs and simply propagates the state estimate using a math model (prior). This causes the estimate to diverge from the true state, or in other words, to deviate from reality. We say the Kalman Filter “has gone to sleep.” There are many possible causes for this divergence, and we don’t need to examine all the possibilities here, but the problem may be summarized as two types. The model used to propagate the prior state estimate forward in time may be too simplistic and needs to be improved, or there is some random element in the model that is not properly accounted for. These ideas, if applicable to cognitive processing, suggest that to avoid a trapped prior, we might consider expanding the dimensions or fidelity of our understanding, or else increase the uncertainty we apply to our prior. The latter solution is obvious and unlikely to be adopted since overconfidence is implicit in the problem of trapped priors. I wonder, however, if we might focus on improving our cognitive models, i.e., the mental model that produces our priors. Perhaps learning more about the dynamics of what we’re fearful of, for example “how do Rottweilers express themselves, and how are they stimulated to aggression or play?” Or, for reducing political demonization, “how do people of other political persuasions prioritize their value judgements?“ This is something psychologist Jonathan Haidt has emphasized and perhaps it is already part of standard psychological desensitization treatment. It seems possible to me at least, that improving our “dynamics model” of the thing we oppose or despise or fear may not meet with the same cognitive resistance as simple exposure to the results (measurement).
It looks like specific phobias (like cynophobia) are rare, for example this article https://www.med.upenn.edu/ctsa/phobias_symptoms.html says all specific phobias combined affect 7-9% of the population. I'm not sure what percentage is just cynophobia, but since most people have experience with dogs in their life and most people clearly do not get cynophobia, this means trapped priors about dogs are rare, most people can have a good and a bad experience with dogs, yet don't get trapped. If political trapped priors are exactly the same mechanism, I don't get why it seems like most everyone has them, instead of just small percentage of conspiracy theorists, etc.
This is a lovely and illuminating piece.
There is a small flaw though on the chess piece illusion. I'm sure the actual illusion is based on the pieces being the same color but the brightness changing the perception. But, the pieces in the photo are actually darker vs lighter just as we perceive them. I pasted out a little section of both Kings lower part and the darker appearing one is actually quite a bit darker in reality, on the screen, compared to the lighter appearing one. Conceptually, though, the point is fine.
How about a stuck prior about a positive emotion? I'm not sure whether addictions qualify, but pursuing a hobby which isn't much fun any more because every time you think about it, you remember how much fun it used to be would count.
(off the cuff) I can't help but think that political partisans on the right, will interpret this post as "force feed conservatives LSD and party drugs until they become brain-addled progressives", with a side dish of "The Soviets were also a big fan of psychiatric treatments for having the wrong politics".
Great article, as always. I think, though, that you may be downplaying the role of self-interest (or at least self-aggrandizement) in this mode of thinking. You focused on politics, religion and phobias, areas where these thought loops seem to kind of “happen to” a person rather than being self-directed (OK, maybe not politics so much). The first thing I thought of, though, were tax protesters, and they don’t really fit the mold. I don’t know where I’m going with this, but that’s never stopped me before. Off we go.
For the uninitiated, tax protesters are a (very) loosely affiliated group of [redacted unhelpful insult] who believe that the federal income tax is unconstitutional and that they are therefore not required to pay it.
There are a thousand different flavors of tax protester, most of which philosophize about the difference between direct and indirect taxation, but the argument that brought them some mainstream credibility was first described in The Law That Never Was, a 1985 book by William Benson (I was hoping to do a book review on it for SSC, actually, but never got it together). The Law That Never Was claimed that the 16th Amendment wasn’t properly ratified in 1913, and that, as the 16th Amendment is what gives the federal government the power to tax US citizens’ income, no US citizen is legally obligated to pay federal income tax.
At its heart, the claim is that the various states that ratified the 16th Amendment ratified slightly different versions of it (so one state might have ratified “The Congress shall have power to lay and collect taxes on incomes, from whatever source derived, without apportionment among the several States, and without regard to any census or enumeration,” while another state didn’t capitalize “States” or DID capitalize some other random word, or used “remuneration” instead of “enumeration”, or a dozen other minor variations). Since three-quarters of the States are needed to ratify an amendment, and nowhere near three-quarters ratified any particular version of the above (both of which are true), the 16th Amendment lacks the force of law.
To attempt to tie this into Scott’s article: I think that if this sounds intriguing or plausible to you, it’s probably because you have a strong prior to having more money, and you’re ignoring the nagging doubts that otherwise would have made you laugh this off the way you would a Seventh Day Adventist. This is already going on too long, but: this argument fails because (1) most amendments were ratified with slightly different language since they didn’t have computers and shit to keep everyone on exactly the same page - if the Sixteenth Amendment isn’t in effect, neither is the First; (2) they considered this back in 1913 and the Secretary of State ratified the amendment anyway; (3) the federal government more or less already HAD the power to tax incomes before the 16th Amendment, so the Amendment doesn't mean what you think it means; and, above all else, (4) you aren’t going to walk into a federal court and convince a judge that the entire federal income tax structure is null and void and the government is required to go bankrupt, effective immediately, so cut me a check.
When I was in law school I got very interested in these sorts of weird legal arguments. Number (4) above is was intrigued me the most: how could anyone believe that this kind of legalistic thinking could have any practical effect? What was the end goal?
So I dug into it, I visited the forums where tax protesters hung out and swapped caselaw minutiae and jerk lawyers showed up to mock them. The community maps to Scott’s outline of “trapped priors” precisely. These were guys (they were almost ALL guys) who walked the walk. They weren’t hypocrites. They were in court making these arguments, despite being directly told by the judge that they were going to go to jail if they kept pushing it. And every time they were warned by the judge, every time another member of their community had their lives wrecked by brutal fines or prison terms, they believed in their cause more. The fact that nobody ever succeeded in avoiding the income tax by making these arguments seemed to reinforce the rightness of their cause rather than convince them that it was a losing proposition. I never really saw anyone change their minds, either through failure or through outsiders to the community explaining reality to them.
This was an awful lot to type when I don’t really know where I’m going with it. Sorry to anyone who read this far. I guess what I’m saying is: while my somewhat-extensive contact with tax protesters has convinced me completely that failing over and over again reinforced their beliefs rather than making them question their priors, the loop was also inextricably tied to self-interest, and would not have existed without it. They wanted money, and they wanted to be fighting a noble fight, in that order. They weren’t bitten by the IRS as a child and developed an irrational desire to fight back. I guess I’m asking: is it worth exploring the degree to which this thought process is actively selfish, and maybe approach treatment from that direction?
I think there's a better way of thinking about this. The political, scientific, and conspiracy examples of trapped priors have a better explanation, which I think is separate from the explanation for phobias.
Everyone has a model of where evidence comes from and how it gets into their heads. If this model is updated to say that evidence from source X is unreliable/biased/adversarially selected, then all subsequent evidence will be interpreted using that model.
If you come to believe that opinion X is supported by made up statistics and bad science, then you get stuck on the belief not-X. Your strong belief in not-X isn't what's causing the problem, the problem is caused by your beliefs about *how evidence is getting to you*. E.g. you believed GMO's are dangerous, and you also believe that any studies that disagree are funded by GMO companies and are therefore false. The way to change these beliefs is not to provide more studies that say GMO's are good. Instead you have to provide evidence that the studies aren't influenced by the interests of the companies.
In this explanation of stuck priors, your beliefs about "how the evidence reaches you" (meta-beliefs?) control how you update your object level beliefs. If humans were less computationally restricted, we would always be able to maintain many different meta-belief models, keep track of the probability of each, and for each meta-belief model we would have to keep track of the object level beliefs that your observations point to, *assuming that model is correct*.
I think the reason humans get stuck on beliefs is that it's *really hard* to keep track of multiple models without letting the object level beliefs formed under each model interact with each other and mess the whole thing up.
The solution to stuck priors, the way to get people unstuck, is to change their meta-beliefs. This is difficult and slow. For example, if a moon landing conspiracy theorist became friends with a few NASA employees, they would probably start to update their meta-beliefs, and get unstuck. But if the friendships were in any way deliberately arranged, then it's easy for the meta-beliefs to explain away the evidence as part of the conspiracy. The way to change these meta-beliefs is for evidence to sneak up from the side, from a source that the meta-belief doesn't automatically discount.
Under this model, there is no amount of evidence that would drown out the prior and cause an update, *if the evidence comes from a source which the meta-belief says is broken*.
This explanation doesn't help explain phobias and bitch eating cracker syndrome. I think they are a separate thing, explained by broken interactions between beliefs and emotions.
This is an interesting idea! However, it seems a lot (to me, as a math nerd) like trapped priors are an implementation error rather than a fundamental feature of Bayesianism. The fundamental reason why is that as more (say) "dogs are actually mostly friendly!" evidence accumulates, the odds ratio for "dogs are friendly" vs "dogs are dangerous" should keep going up, because Bayesian updating is basically multiplying the prior odds ratio by an odds ratio representing the weight of the evidence.
Related idea: the conservation of expected evidence (mentioned - introduced? - by Eliezer Yudkowsky at https://www.readthesequences.com/Conservation-Of-Expected-Evidence); what you expect to believe in the future is what you believe right now, if you're doing Bayescraft correctly. So if you were some person with a "dogs are dangerous" trapped prior, and you forecasted that future!you would believe that dogs are dangerous even more than present!you *no matter what happens*, then you should already have that level of belief in the dangerousness of dogs.
(The rest of the problems - like having beliefs about how dogs lead to unpleasant experiences, or how Democrats are morally evil - can be avoided by only holding beliefs about unambiguous object-level things that are outside your control, which avoids both ideological ambiguity and fixed-point problems/markets for entropy/etc.)
I'm confused/skeptical on three counts.
First, you've said in the past (e.g. in "Going Loopy" and "Can You Condition Yourself?") that the brain is in general pretty good at filtering out its own contributions when predicting things; it's not clear why this case is an exception (in the phobia case, I assume the answer is 'the thing that distinguishes *-phobia from normal fear is the failure to filter correctly', but if you're using stuck priors to also explain confirmation bias, then that's solidly in the domain of normal, non-pathological reasoning).
Second, why does the gradual version of exposure therapy work? In normal reasoning, a photo of a dog provides no evidence about whether I should be scared of dogs, a dog in a cage provides very little evidence about whether the dog would be dangerous outside of the cage, and so forth. Given that the cynophobe is supposed to be less responsive to evidence than normal, why doesn't that all get rounded down to zero difference from the prior, the same way they round away the much stronger evidence of being in a room with a Rottweiler and not getting bitten?
Third, the Drummond and Fischer paper finds that polarization increases on issues to do with political and religious identities, but not on issues that aren't politicized. That seems to me not like a trapped prior (if I can have a trapped prior about non-political issues like phobias, presumably I should also be able to have trapped priors about arbitrary scientific beliefs); on the other hand, it's perfectly consistent with a signalling/group identity type of explanation. Signalling also seems like a better fit for dog whistles; if I have a prior that Ted Cruz is malicious and he says 'I'm against New York values', then I might naturally update to 'New York values are good', or 'New York values are bad and Ted Cruz is lying about being against them', but 'New York values are a coded message for Jews, and Ted Cruz is conveying his anti-Semitism in a way intelligible only to his base (and also his opponents currently interpreting the dog whistle)' seems like a stretch. On the other hand, if I just want to make it really clear to everyone that I'm part of the Ted-Cruz-hating-tribe, then making an argument about secret codes does a much better job than providing a nuanced critique of one of his policies, because I might disagree with his policies for reasons other than partisanship, but I probably won't believe dubious theories about signals for non-partisan reasons. (Likewise, if I want to signal my partisan affiliation on the pandemic, I'm not going to look at a hard question like 'how much economic damage is worth it to mitigate spread'; I'm going to say something totally unhinged like 'Outdoor gatherings are a public health risk if you're having fun at the beach but totally safe if it's a BLM protest' or 'wearing a mask is about authoritarians conditioning us to submit to their orders' precisely because there's no non-partisan way to misupdate on evidence badly enough to reach those beliefs).
Somewhat related, in machine learning there is a notion of a “learning rate” which you can think of as the amount by which you update your prior based on new information. In the Bayesian outlook there is a precise right value for this parameter, but in practice people play around with it.
Interestingly, it’s often much better to underestimate it than to overestimate. If you underestimate the learning rate (which results in a tendency to stick to your priors) then you might not be as efficient with your computation or samples, but eventually you will reach the right point. In contrast, if you overestimate it, you may well never converge.
The above might be why it’s a decent heuristic to err on the side of under-correcting priors than over-correcting.
In the ML analog, trapped priors would be like a bad local optimum that we cannot escape from without making a large change. This also can happen sometime.
For many of these, it seems like there's something else going on--some kind of tossing out of outliers. If you have normally-distributed data and see some value that's +10 sigma out, it's almost guaranteed to be measurement error or something. But if your model is "normally distributed data," but the actual source of the data isn't really normal but instead some weird distribution that allows for +10 sigma events to happen pretty regularly, you'll filter out the problems with your model as being errors or lies or something.
When you tell me you saw a meteor shower last night, or a bad accident on the road today, my model says that's plausible, so I assume you're probably telling me the truth and may update my understanding of the world based on that. When you tell me you were abducted by aliens last night, or saw a giant moth flapping its wings over the city today, I'm probably just going to conclude that you're delusional.
When you make a plausible-sounding argument for medicare-for-all or shall-issue laws, I probably update my internal understanding of the world toward those being sensible policies. When you make a plausible-sounding argument for sacrificing babies to Moloch or re-imposing slavery, I probably update my internal understanding of whether you're a nutcase whom I should ignore instead of my understanding of whether maybe there's a good argument for baby-sacrifice or reopening the Atlantic slave trade.
In all these cases, what's happening is that your evidence/claim/argument is updating two different parts of my model, the "is the object-level thing they're claiming true?" and "are they a good source of evidence/claims/arguments?"
I don't think you really discussed it in the post, but an important characteristic of the types of scenarios that lead to trapped priors as you describe them is abstraction of goals. For example, the goal of a fear response to a dog is to avoid getting mauled by a dog. When you gather data from experiences with dogs, the thing your brain should* care about is whether the dog mauled you. But instead of updating P(I Get Mauled | I See A Dog), your brain updates P(Get Terrified | I See A Dog). So your brain is updating on an outcome that is, in a sense, one level abstracted from the thing it "should" care about.
This misalignment appears critical to to the phobia example, but I'm not sure if the same model makes as much sense in the politics examples. Using the same construct for some Policy X that my side supports, we'd have the brain updating not for P(Policy X Is Good | Evidence About Policy X), but instead for P(My Party's Policies Are Good | Evidence About Policy X).
I'm not sure how intuitive that sounds, which makes me wonder if the way the abstraction works in the two examples is different.
*at least in some sense
I'm not fully convinced that trapped priors can happen as a natural side effect of Bayesian reasoning. That is, I'm not convinced that purely "cognitive" trapped priors are possible.
As you note, when Bayesian sees evidence for X, they should never update in the direction of not-X. If you think that bad things are 90% likely to happen whenever you interact with a dog, and you have an interaction with a dog where nothing bad happens, you should update the 90% number to *something* lower.
Maybe only a tiny bit lower. But if it goes *up* afterwards, you simply aren't doing Bayesian reasoning anymore.
Perhaps you're doing some approximation to Bayesian reasoning that has this failure mode. But if so, it matters which approximation, and how it causes the problem -- if the approximation is doing all the work of explaining the pathology, then the treatment ought to focus on whatever is wrong with the approximation.
------
Related: in the dog phobia example, I think you may be conflating two things
- "how dangerous a typical dog interaction is likely to be"
- "how dangerous that particular dog interaction actually was"
The former is what the prior is about, and what gets updated to form the new prior. If you're Bayesian it should always *move* in the right direction.
The latter doesn't move, you just conclude whatever you conclude. It can *point* in the wrong direction, i.e. belief = dangerous while reality = not dangerous. But this one example doesn't become your entire prior for future experiences. Your prior is just whatever it was before, plus one update on a non-dangerous dog.
------
I want to connect this with jeff wade's comment about Kalman filtering. Kalman filtering really *is* an instance of Bayesian reasoning that can get off track and never really converge with the evidence.
What's the difference? Kalman filtering is about estimating a changing state, not a static fact like "how dangerous dogs are are a species." You predict the next state from a dynamic model, you make noisy observations, and you combine your prediction with the observation to estimate you actual state.
Here, the *state* gets updated, but the *dynamics* does not. If you predict "things get worse," and things look better instead, you will update to "things are currently [less bad than I expected, but worse that they look]." However, you don't update towards being less pessimistic about the future; you'll keep making pessimistic predictions and then partially correcting them, forever.
In the account of belief updating in this post, it sounds as if the prior gets updated twice -- first you combine prior and raw evidence to determine "what happened," then you combine "what happened" and your previous prior to get your next prior.
This is not how Bayesian inference works when you are estimatic a static fact. However, it's kind of close to what Kalman filtering does to estimate a changing state. The first step is normal Bayes to figure out where you are right now; the second step takes where you are now, and projects where you will be next.
I think that maps best onto human cases of continued engagement with one individual, like the example of abusive relationships. In such a case, someone could keep predicting "things will get worse" and getting corrected back to "things are the same," but never corrected towards more optimism. It might work for political groups if the belief is of the form "the Republicans are always getting worse" rather than "the Republicans are bad" It probably doesn't work for dogs.
(Hmm, maybe it's that every *individual* interaction involves some dynamics. Like, you always think the dog is going to get really mean in the next 60 seconds, and every 60 seconds of benign caninity makes you update to "it only got *slightly* meaner," and that still adds up over time.)
This seems like an especially good post to cross-post to LessWrong. I would love if you crossposted more in general, but this one is especially fitting.
Probably a dumb question: Why does the brain work that way?
Like, suppose you have sensations S1, S2, ..., S10 each of which is mild evidence against a trapped prior P1. According to this model your brain reasons from S1 + P1 to an unchanged (or stronger) prior P2, then from S2 + P2 to P3, and so on. And none of the sensations is enough evidence to overcome the prior, so things just get worse.
But in any reasonable framework, S1... S10 together are stronger evidence than S1 alone. And with enough sensations that should be enough to overcome P1. So the correct Bayesian math would be for the brain to update P1 + S1, S2, ..., S10 to a lower prior P'. In other words it combines its original prior with all the evidence since that time.
Why can't the brain do that? In algorithmic terms, is it a complexity / memory / "storage" limitation?
I guess one explanation might be that we don't actually remember sensation, just perception. (Partial because it begs the question "why not?") But that's hard to square with the political applications-- do we really "not remember" evidence regarding our political beliefs?
"...should try to identify a reliable test for prior strength [...] then observe whether some interventions can raise or lower it consistently. Its goal would be a relatively tractable way to induce a low-prior state with minimal risk of psychosis or permanent fixation of weird beliefs, and then to encourage people to enter that state before reasoning in domains where they are likely to be heavily biased."
Isn't it currently a Dark Epistemic Art? That is, to see the best available method, shouldn't we just check how people that succeeded in convincing others of really weird shit do it, and then try to replicate the results but without pre-determining the conclusion?
I think that the ideas you've presented relating depression to placing low priors on sensory evidence might explain the apparent high rate of trapped priors in politics.
We have a word for people who place a low prior on sensory evidence: rationalists. The word is Latin, but the division of the Western world into people who trust sensory evidence (empiricists) and people who don't (rationalists) goes back to Thales and Parmenides.
Rationalists always conceal the existence of this division when they dominate public discourse, and we're in one of those rationalist-dominated periods today, despite the continuing successes of empirical science. So today, most people think "reason" and "rationality" are synonyms, when in fact "rationalism" is an oversimplified caricature of reason which states that:
- the world is split up neatly into categories by the words of our language
- every statement expressed in those words is either True, all the time, everywhere; or else it is False, all the time, everywhere
- Reason means beginning with a set of axioms or divine revelations, then applying logical deduction in order to deduce new Truths
- you can believe anything you've deduced with 100% certainty
Empiricists approach politics as a practical matter, in which we have a limited amount of resources to allocate in a way that will optimize some measure of utility. Rationalists approach politics as a moral matter, in the mindset of Plato when writing Republic.
It so happens that, today, every political party is rationalist. (This is usually the case. Empiricists are usually out discovering stuff, making stuff, or trading stuff, rather than playing politics. Democratic Athens, the Republic of Venice, the Netherlands and some Italian city-states during the Renaissance, and the American Revolution are the only empiricist states / political movements I can think of at the moment.)
- All Christians are rationalist, because Orthodox theology, and to a lesser extent the New Testament, were based on Plato.
- All Hegelians are rationalist, because Hegel was very Platonist. (Even if you disagree, you still gotta admit he was a prototypical rationalist.) This includes Nazis, Marxists, and Progressives. "Progressive" doesn't mean "a person who wants to make things better"; it means someone who believes in the divine march of Progress, under the guidance of the World Spirit, as explained by Hegel, to achieve the ultimate "perfection" (a Platonic concept) of the world.
- Libertarians are rationalist, because they have one supreme value (liberty), which takes priority over all other values, at all times and in all places.
- For the same reason, environmentalists, feminists, post-colonialists, and almost any "one great cause" movement is rationalist.
Rationalists are by definition people who place low value on empirical evidence. This is why Christians still won't believe in evolution, Marxists don't care how many times Marxism fails disastrously, and the "defund the police" movement doesn't care that excluding the police from the CHOP in Seattle for 3 weeks made it the most-violence-prone place in all of America for those 3 weeks. Theory has epistemological priority over observational evidence to rationalists.
> For example, in the 2016 election, Ted Cruz said he was against Hillary Clinton's "New York values".
Actually he said he was against Donald Trump's "New York values". Hillary Clinton had nothing to do with this statement; it was made in the sixth debate for the 2016 Republican primary. https://time.com/4182887/ted-cruz-new-york-values-donald-trump-republican-debate/
Mechanistically, trapped priors may be related to the phenomen when memories become experience. So, one spend time with a dog, the conclusion that it's terrible is confirmed. After that one recalls the experience, and brain uses the posterior+real experience to generate new "memory" which replaces experience. This biased memory is then combined with previous priors to give new and worse posteriors (which becomes prior next time one recalls the events). And so every time one recalls something terrible, it becomes even more terrible. And if the event is "triggering", the recall/reconstruct cycles are accelerated, one thinks about terrifying events much more frequently, and the process of replacing real memory with biased ones is very fast.
The description of trapped priors sounds a lot like Lewis's reply to Hume on miracles: having decided a priori that miracles are impossible, Hume dismisses the (rather large) volume of testimony about miracles. He cites the credulity of the witnesses as evidence against their reliability (why credulous? c'mon, they're talking about angels and demons and stuff!), and then concludes from the absence of evidence that miracles are impossible.
Make of Hume's argument what you will, but it's a striking structural similarity to the trapped prior phenomenon.
I'm quite skeptical of this post.
I studied neuroscience in undergrad. I recall the brain having fairly idiosyncratic machinery for... well, everything. Including fear and anxiety.
Like if you were going to *design* a brain with an anxiety-processing center called an amygdala, you'd probably figure out what inputs the amgydala needs and just... give it those inputs. But no, the brain takes perceptual information and sends it to the amygdala via two different pathways: a fast path that goes straight to the amygdala, and a slower path that goes first to the cortex and then to the amygdala. (https://en.wikipedia.org/wiki/Joseph_E._LeDoux#Work_on_threat_response,_anxiety,_and_emotions)
Does this have implications for fear and trauma and memory? LeDoux thought so! (Though my information is dated now.)
My point is, I don't think the brain is "just doing bayesian updates." I don't think that all neural processing is to predict things and then update those predictions in a bayes' theorem-esque manner. (I know Scott has written about this in the past. I'm not sure to what extent he thinks this is literally true.) I think that the brain is messy, and that no matter how the low-level computation works there is some really weird and idiosyncratic "programs" that are built on the low-level stuff, and at least some of that is involved with fear and looks very little like Bayes' theorem.
I think this post goes too far in the "everything is Bayes" direction. It's an attempt to unite disparate observations, which is great, when it works. I don't think it works here. It seems too pat. And not very predictive.
----------------------
Those are my overall thoughts. I also have a grab-bag of random things that came up while I was reading:
1. This post seems to imply that a lot of biases can be described as "bayesian reasoning + up-weighted priors." You've got: bias in favor of one's tribe, self-serving bias, and I imagine that "bias in favor of thinking good things about oneself" would also be included.
That said, what is this post meant to explain? If the theory doesn't logically entail that humans will have *particular* biases, then it's not very helpful. Why do people have "trapped priors" when it comes to their tribe? Why not some other thing?
Also, if "trapped priors are purely epistemic and don't require emotion," why are all of our good examples of trapped priors related to emotion? Maybe there's more going on here that this simple theory can't explain.
2. If the theory is meant to suggest ways to "unstick" sticky beliefs (say, by using psychedelics), do we expect psychedelics to affect all of the biases that (we claim) result from trapped priors? Will people show less self-serving bias? This strikes me as predicting too much.
3. A number of people in the comments are now saying "I have a trapped prior for thing X," meaning, "I have trouble changing my thoughts/feelings about thing X."
But this is post hoc. The trapped prior isn't predicting anything. "Trapped prior" is becoming a name for a thing, but not a *cause* for a thing.
This is, obviously, not Scott's fault. But the point should be made that we need independent evidence for calling something a trapped prior, and the fact that we don't have that is currently a weakness of the theory.
4. "...you only habituate when an experience with a dog ends up being safe and okay. But being in the room with the Rottweiler is terrifying. It's not a safe okay experience."
By this logic, it would be very difficult for the rats in your example to habituate to the bell, wouldn't it? The bell would cause them to predict pain, just as the dog causes some people to predict pain.
(Yep, I studied neuroscience in undergrad and an explanation like Scott's made sense to me at the time. Now, I don't really think it does.)
5. "...[psychedelics] can loosen a trapped prior, causing it to become untrapped..."
What does "loosen" mean in statistical terms? Push a prior toward uncertainty (.5)?
This question is less because I want an answer, and more because if everything comes down to Bayes theorem, then it would be helpful to *use simple terminology that applies to Bayes theorem* when talking about this theory. "Loosen" is very vague.
Not really the point of the post, but I think it’s very plausible to think that joe Biden personally doesn’t support rioters but most people in his administration — starting with Kamala Harris — do. How else would you interpret tweeting out a bail fund for rioters?
I think that vanishingly few political disagreement are actually about facts, so it's hardly surprising that evidence about facts is unlikely to change them.
Facts are tiny side-issues that people spend a lot of time arguing about, but which are usually pretty tangential to the point. If Alice and Bob sit down to argue about the death penalty, then Bob can bring a whole bunch of studies that show that the death penalty reduces crime rates, and Alice can bring a whole bunch of studies that show that the death penalty actually has no measurable effect on crime rates, and they can both spend many hours arguing about the relative merits and flaws of these piles of studies.
In the end, though, this debate is all for nothing -- Alice disagrees with the death penalty on principle, and even if you convinced her that the death penalty _does_ slightly reduce crime rates then she's not going to change her mind, she'll say that a small increase in crime rates is a small price to pay for the abolition of that horrible barbaric punishment. (Nonetheless, she will fight tooth and nail against having to admit that it actually _does_ increase crime rates.) And same deal for Bob.
Or, if you listen to libertarians, you will hear a great host of arguments about how getting the government out of such-and-such would lead to improved outcomes. I have never once heard a libertarian say "Look, to be honest, government interference in XYZ actually probably has positive outcomes, but I still think that the government should get out of XYZ on principle".
Are increased minimum wages good or bad, overall? It doesn't matter, because everyone seems to either support them on principle or oppose them on principle.
Why do people insist on debating facts when their disagreement is about values? Possibly because they are arguing for the benefit of some hypothetical listener who might have no particularly strong value opinion on the matter but are pragmatically inclined towards good outcomes. And maybe these people do exist out there in the middle somewhere, but they tend to be relatively quiet. Among the sorts of people who debate politics on the internet, though, meaningful fact-based disagreements are practically absent.
BECS is probably the thing that generates the most ill will. As I'm on the right, I constantly hear that so-and-so - or I - is racistsexisthomophobe because of innocuous comment X. I don't think there is anything so guaranteed to cause ill will as "Oh, you _really_ mean X when you say Y."
(That said, I've tussled enough with White Nationalist types to be aware of Scott's point on Weak Men.)
The best thing I can think of is that, even if you think this, not to say it. Try and engage with the literal meaning of the words as best as possible (the late Hitch was good at this). And if you must say "This looks like X to me", explain that it is something you suppose, and why you think it, not just say "You really mean X".
For example, I've dealt with a lot of people who have nothing good to say about Israel, and people who have nothing good to say about the new South Africa. And who in both cases offer euphemisms about the terrorist movements each face (yes, S.A. has its white nationalist terrorists; they are called the AWB). Now I cannot say that each of these people is anti-semitic/racist, but what I do say, if I hear enough of this, is to say "Look, this is giving me a really bad feeling, and here's why."
That's the best I've come up with at the moment.
So there are multiple ways to have trapped priors? Let's take a group of people that have a strong trapped prior that dogs are dangerous:
* Alice suffers from a weird sensory disorder that makes dogs look really dangerous.
* Bob suffers from a weird brain disorder that makes him unable to update his prior with new information.
* Carol has cynophobia. A dog "is so scary or hated that it's aversive to have to perceive it directly, [Carol's] mind decreases bandwidth on the raw experience channel relative to the prior channel so that [she] avoid the negative stimulus."
* David hates dogs for political reasons. Once again, a dog "is so scary or hated that it's aversive to have to perceive it directly, [David's] mind decreases bandwidth on the raw experience channel relative to the prior channel so that [he] avoid the negative stimulus."
Can we think of more cases? How much does these four cases really have in common? What is the actual difference between someone who has a strong prior that dogs are dangerous and someone who has an equally strong but *trapped* prior that dogs are dangerous?
Perhaps we need an easier approach. For example, I have found it helpful to ask students, "Are you willing to accept that your idea is wrong if you are shown evidence that it is wrong?" Prime the person to overtly and rationally accept the possibility of change. Have you or other workers in this area tried that?
Also, you are looking at behavior change, but there is a pretty solid body of evidence that behavior change is more enduring when the new behavior occurs at higher frequency with less latency. You and others ignore frequency, as far as I can tell. If you want people to change responses, it might be more effective to train them to change a small response quickly and then do it repeatedly. Stop looking only at quality and look at frequency.
I will give you an example. If you want your client to stop feeling intense fear of dogs, do something to help her quickly change the feeling. For example, give the person a picture that causes a fear reaction, then teach her to change that feeling quickly. I don't know your clients, so I don't know what would do that, but you could possibly give a mild fear stimulus and then very quickly show a picture of a dog or some non-doggy but delightful, relaxing, or humorous stimulus that would quickly alleviate the feeling of fear. Alternatively, you could teach a mental reaction that would relieve the fear and teach her to quickly engage this reaction (e.g., a statement of some kind). Then have her do this more and more quickly for a minute or so, using multiple stimuli and trained reactions or competing stimuli.
This is just an outline of an approach that might work. The main point here is that frequency and latency are both important, and as far as I can tell, you are not much aware of that.
Here's my response to this post https://www.lesswrong.com/posts/JtEBjbEZidruMBKc3/are-dogs-which explains my view on how one should perform the updates on dog encounters, and what can go bad if you let the final verdict influence your (meta)priors.
I agree with most of your analysis, but I've met a whole bunch of people for whom "punish women for being sluts" was pretty much the explicit reason of choosing a pro-life position. Typical mind fallacy.
From the mathematical point of view it seems to me that there are two intertwined but different issues.
1) You start with a prior that is extremely narrow, a Dirac delta for any practical purpose, or maybe just something with a finite support (i.e. zero outside a certain interval). This means that even if the Bayesian updating works perfectly you can't escape the (for all practical purposes infinite) prior information that you start with. By the way "prior with finite support" is what came to my mind when I read the phrase "trapped priors".
2) The updating algorithm deviates from perfectly Bayesian and can actually update the prior in the wrong direction (it could also be Bayesian at the core but with some inbuilt "data preprocessing" that amplifies evidence in one direction disproportionately).
Maybe a similar point has been raised in other comments (I haven't read them all)... Anyway it's a probably superfluous "technical" comment that doesn't change the take-home message of this brilliant analysis.
Hah, we just covered this in a course I'm taking. Our model was like this: a queue of people have a prior that restaurant A is better than B with 51% probability. Now each person on the front receives a signal that one of the restaurants is better than the other with 52% probability.
If the first person receives the signal A>B, then everyone in the queue will end up going to A: When they're at the front, they have their prior, their signal and they can infer the first person's signal is A>B (since they ended up going to A). Combining this information will always yield A>B regardless of their own signal.
"Trapped prior" is a good way to describe Scott Alexander's attitude towards the New York Times. The sensory evidence (the NYT being full of people wanting to destroy Scott Alexander's life and not afraid to lie in service of that goal) has been swamped by the prior (the NYT is the font of all that is true and holy)
false
Alex Gjust now
I feel very skeptical about entirely-internal-belief-feedback-loops but more open about a mixed internal+external-belief feedback loops (or local minima in belief-space which, say psychedelics might help with)?
Say, if you are a partisan Democrat and have a discussion with a partisan Republican probably the interaction will be very bitter and shouty and you will develop more of an ugh-reaction to Republicanism
Consider the hypothesis "you are in a russian roulette environment." Ie an environment that has some small chance of killing you.
So immagine your priors say, either dogs are safe and quiet, or dogs bark a lot and occasionally kill. Then when you see a barky slobbery dog, your belief in their danger goes up. Maybe the hypothesis "dogs are barky but safe" is in there too. If the chance of dogs killing you in the dangerous dogs hypothesis is 1%, thats going to take many 100s of very scary dog visits before it shrinks away. (Enough that you would expect to be dead if dogs were that dangerous)
So with these fairly sane priors, and perfect baysian updating, you get the behaviour where they walk away from a dog uninjured, but are more scared of them.
Of course, many things look baysian with strange enough choice of priors.
The doomsday cultists. Ok, they are just not baysian, but that could be a lot of selection there, in that the least rational are most likely to be doomsday cultists.
Don’t be so down on cognitive trapping; it can have big benefits too! It’s fast (all precomputed, no serious thinking needed in the hot path in response to stimuli) and it’s immune to lyin’ liars (because no matter how eloquent they are you’ve presupposed their falsehood).
And there’s all sorts of reasons you might want fast and durable mental tools. You want them if a tiger is hoping to eat your face. You want them if an abuser is trying to gaslight you. You want them if you’re being tempted away from the high road. All things where you don’t have time to think and it costs a lot to be wrong.
It’s sort of the opposite of the scientific method, which is all about focusing on exactly what stimuli should flip your prior. But science sometimes takes decades, or lifetimes, or more. Time enough to cut through the noise.
The other way through cognitive trapping, I believe, is humor. My father taught me that the way to defuse any situation is with a gentle joke. Perhaps if you’re laughing with everyone then you can’t register the experience as negative? And therefore you can begin to update your prior in a more Bayesian fashion? I do get the sense that my father’s use of humor has more to do with him being a middle child and less to do his neuro PhD or his pediatric neuro MD, but I’ve come to believe in his method.
Aside: one of the really fascinating things about working with geniuses - not just very smart people but geniuses - is that they develop a strong cognitive trap that most people are wrong and they are right. It’s not totally irrational! Except when it is.
An interesting analogy to this issue might be "regularization" in machine learning (neural network) frameworks. To prevent over-fitting (i.e. a network that learns irrelevant patterns or "noise" in its training set), data scientists optimize a fitness function that includes some features of the network itself.
One common regularization option is L1-norm regularization, which penalizes the network's fitness score by the sum of the absolute values of its weights. This normalization encourages networks to push as many weights as possible towards zero, forming a sparser (fewer-factored) explanation for any inference.
Shifting back to "trapped priors" and using this as a metaphor for human reasoning, habituation requires us to replace a simple explanation ("all dogs are scary") with what is at first a more complicated one ("almost all dogs are scary, but this dog isn't scary because it's licking its own rear end right now.") If we suppose we're instead applying a strong preference for a simple rule, then a simpler explanation that's less accurate ("all dogs are scary, and also this dog is scary") may still be preferred. Habituation would then work because the strong belief is not directly challenged (even if all dogs are scary a *picture* of a dog can't be scary), so the preference for a simple belief is not triggered in the same way.
Isn't this as simple as "Priors should be falsifiable"?
> I can't unreservedly recommend this as a pro-rationality intervention, because it also seems to create permanent weird false beliefs for some reason, but I think it's a foundation that someone might be able to build upon.
It creates weird false beliefs because lowering the prior doesn't remove the error.
There's a type-I/type-II error tradeoff here. Stick more strongly to (typical) priors, and you'll be less of a conspiracy theorist but more vulnerable to failing to notice that the building pandemic really is that bad. Stick weakly to priors, and you'll be quick to panic about COVID in January 2020 but also become convinced that lavender cures autism or whatever. The larger the errors in the inputs, the more important this tradeoff becomes.
Since my whole mission here is to figure out grey-tribe failure modes, here's a speculative one: the grey tribe likes to think about weird ideas because it thinks bad ones will fail and good ones will succeed. Smart grey tribers have relatively low error in their analysis, but the error is heterogeneous, and if you consider enough weird ideas you will eventually land on one where your error convinces you of a false premise strongly enough to lock. And the tradeoff can't save you: low-prior people are more vulnerable to the error, and high-prior people are more easily locked into it.
You've written about autism/schizophrenia being at each end of the evidence/priors spectrum. Since schizophrenia is "the one with the hallucinations" (sorry for this terrible oversimplification) I find it weird that psychedelics, "the thing that causes you to have hallucinations" (again, sorry), map onto the 'evidence' end of the spectrum.
I think the problem is that there are two fundamentally different things that superficially look like giving more weight to evidence:
1. One is actually upregulating the evidence channel vs the priors channel. EMDR and mindfulness may work this way. Seeing pulsating objects and auras while tripping also seems to go here. Because your priors are downplayed, low-level quirks of your visual system come through unaltered. Evidence is given more weight, so colors appear more vivid and saturated.
2. The other phenomenon is akin to increasing temperature in the annealing metaphor. Because higher temperature makes the prior landscape effectively flatter, you need less evidence to get out of local optima, but it also means you jump around a bit. So, even if you don't particularly give more weight than usual to your senses, it may be easier for a cloud to induce a fleeting experience of pareidolia - and it's gonna be equally easy for you to snap out of it and fall into a different, random attractor state, and then another, and so on until the LSD wears off and your prior landscape looks rugged again.
To sum up: in this model, autism and schizophrenia would be variation along the first dimension and independent of temperature; psychedelics would be affecting both knobs, making you pay more attention to evidence _and_ making your priors more malleable.
Not sure any of this makes sense, but at least I now have a better understanding of my own confusion :)
I believe the framing a person applies to an issue is capable of overriding everything else, and that this is the most common way trying to be an educated, informed person goes wrong. This is not only a trap; this is THE trap, as long as you are human. I'd say most important sources of stupidity that originate in otherwise brilliant people are just manifestations of them framing or contextualizing information in such a self-serving, or, as you formalize it, in a prior-confirming way. I think almost everyone has a natural tendency to do this, especially with emotionally charged issues. (There are simpler mistakes, but those are fixable)
I think a primary exercise of having a healthy perception of the world should include ensuring that one remains humble and attempts to question their most entrenched worldviews as an active practice. This probably includes a mixed information diet from sources that have many different perspectives.
Maybe constantly trying to undermine one's own worldview from all sides is actually the most essential practice if you want to have the best chance of being correct about things. This could be the real central praxis of rationality, even more so than dealing in certain mathematical formalisms (which, while great, can easily be used in service to one's ideology)
Supporting the idea of trapped priors independent of any emotional component, you can very easily end up with a trapped prior in a naive computer implementation of a Bayesian filter that uses normal likelihoods instead of log-likelihoods: if your prior becomes small enough, the floating point value will underflow and turn into zero.
In theory, it can happen with log-likelihoods as well, as IEEE floating points can become infinity if sufficiently large values are added together. However, the amount of evidence you need to gather for this to occur is quite obscene.
Have you considered a trapped prior might actually be an extremely useful tool to have?
Suppose the trapped prior is 'everything will be ok'. Whenever a situation seems to be not ok, if that prior really is trapped, you'll rely on it, which will make you feel that everything is, indeed, OK, and as a result, you get more evidence for the trapped prior.
This seems like a pretty decent description of what religious faith gives people. If you have 100% confidence that things will be OK, that they happen for a reason, and your job is to always accept your circumstances and do the best you can, then you can likely interpret any situation - even ones most people would say are very awful - as being OK, so long as that prior is strong eonugh.
This is the approach i've been taking the last few months, and it seems to be working for me: https://apxhard.com/2021/01/18/the-hypnotoad-pill/
The only hing I don't use this prior for is to anticipate that there will be no bad consequences from a possible course of action. It would obviously not be good to say, "i won't buckle my kids' car seats because nothing bad will happen." I don't' think most religious people are doing that, but i can see how this sort of thinking would not be great at convincing people they should wear a mask or want to reduce carbon emissions, for example.
But if you're crippled with anxiety and fear about the world, a trapped prior that says "i will be ok" seems like an extremely useful tool.
"I don’t know why this doesn’t happen in real life, beyond a general sense that whatever weighting function we use isn’t perfectly Bayesian and doesn’t fit in the class I would call “reasonable”."
Assuming the model, it seems to me that the reason for the problem is that there's a cycle. In your diagram, the gray box isn't where the Bayes happens. It's just some sort of weighted average function between experience and priors. The Bayes happens between the perceptions and the priors. But since perceptions are modulated by priors, this is factoring your priors into your evidence, which is a little silly. After all, it means you can update on nothing: take a 0 experience, average it (in any way) with your prior, and you'll have a non-0 perception to update on.
Most of the time, this probably is fine, because it just makes you update slightly less or more than you should, and in some cases could be helpful (for example, if, say, your brain isn't capable of making large enough jumps in prior in a single bound, then having your prior reinforce itself lets that happen over time, maybe). In some cases, it'll cause problems, like if the average between a large negative prior and a small positive experience becomes a small negative experience, the opposite of the actual experience. So, a proper Bayesian would have to do the updates on prior and experience *before* funneling that into perceptions. Or maybe do updating and perceiving completely separate. At least, the perceptions part would not be an input into the Bayes function.
WRT the "cynophobia" thing, what about cases where dogs truly do behave abnormally aggressive towards a person?
I've long since lost count of the number of times some person, innocuously walking their innocuous dog near me, has ended up totally shocked as the beast goes into a rage, straining at the leash trying to reach me and attack me. They always apologize profusely, saying the dog *never* behaves that way. Well... for me they do, pretty darn consistently.
In a similar vein, I have never in my life met a mean cat. Not even ones whose owners warn me about how mean the cat is. I can always get them to cuddle up in my lap and purr as I pet them with no trouble. Again, this frequently surprises the owner; watching the responses of others is a good way of getting outside my own biases and knowing something real is going on.
Maybe I just smell like a cat on some pheromonal level or something?
I guess my question would still have to be why this actually happens. Is there just too much friction in the system? Is the brain using some sort of computationally efficient approximation of Bayes Theorem that can get caught up on things like this?
Conspiracy theories are another case of this, where they can fully explain the lack of evidence ("because it's a conspiracy!") and become trapped.
More broadly, priors are not just a number -- they're a number attached to a whole mechanistic theory about the world.
If I tell you an urn contains a red and white ball, and then I pull a white ball out of it, then when you see the white ball, your confidence that the next ball will be red goes up.
If I tell you I filled the urn with balls of a single colour, then when I pull a white ball out, your confidence that the next ball will be white goes up.
The same observation, but you update in different directions depending on your prior!
The issue with trapped priors might not be that the updating process is irrational, but that the prior itself was way overconfident to begin with. A properly ignorant prior will contain some seed of doubt in any conspiracy theory, which grows faster on any evidence of no conspiracy than the conspiracy theory's cover-up likelihood. But if you don't have that seed of doubt to begin with, a prior of zero still updates to zero no matter how much evidence is in its favour.
While we're at it, consider how you update on the evidence in these cases.
People say economists are always predicting stock market crashes that don't happen. (Until they do). So some people start to treat the history of past wrong predictions as evidence that decreases their confidence in any further predictions. And I can't say they're wrong to do so.
But what about earthquakes? After a seismologist starts to worry about a build-up of stress in a fault line, and sound a warning about a potential earthquake, each passing year in which that prediction rings false might *increase* the seismologist's confidence that the earthquake will happen the following year. And I can't day they're wrong to do so either.
This is because their model is of a stress value climbing and climbing, which triggers at a threshold that is known to very low condidence. When you start to think that stress is within your trigger range, you start to worry about an earthquake, and when it doesn't trigger you worry even more.
But that's also, more or less, the model that economists use to predict crashes.
You don't need psychedelics to get permanently trapped weird beliefs. Claiming that they are false, however, seems excessive. I have a friend who believes in big foot. He seems to be oblivious to the lack of evidence. But then *I* don't believe in big foot, so my evaluation of the evidence is different from his.
FWIW, I tend to think of psychedelics as something that relaxes the error checking protocols of the brain. If these "trapped priors" are a manifestation of the error checking gone awry, then the whole thing makes sense. But note that the error checking exists for a valid reason. That the mechanism isn't perfect doesn't mean it isn't important.
ISTM that your line of reasoning would indicate that the best way to convince someone with a trapped prior is to present them with lots of REALLY weak (and unemotional) arguments that they are correct. I've heard this approach suggested before, but it's really difficult to intentionally create weak unemotional arguments that you believe are incorrect.
One thing I'm noticing here is that all the priors you mention seem to be beliefs about the believer's own identity. A Democrat can't believe Republicans are right because they believe that they (a Democrat) are right. The cynophobe, at some point, isn't afraid of dogs any more, they're afraid of themselves--specifically, their own reaction to dogs. The cultists ARE reacting to evidence when prophecy fails and they double: they're reacting to the evidence that faith has been rewarded with acceptance within the cult, and a strong counterexample to the cult's belief system is an opportunity to demonstrate greater faith, thereby gaining greater acceptance. There's a reason faith being rewarded with acceptance is part of the memetic code of many religions: it's an effective "immune system" which defends the religion against outside evidence.
The coyote/polar bear example is a harder stretch, but as someone who was rewarded repeatedly growing up for "being smart", I definitely have some beliefs about myself which cause me to react negatively if ANY of my priors are challenged, simply because I've been conditioned that acceptance comes from having correct priors. I think this is a fairly universal cause of irrationality, but those of us who have been rewarded for being right a lot perhaps are the worst afflicted.
The interventions you mention fit this pattern to some extent: putting the cynophobe in the room with the Rottweiler immediately addresses the fear of the dog but does nothing to address the fear of the cynophobe's reaction to the dog--the slower exposure addresses the cynophobe's fear of their own reaction. Psychedelics are correlated with decreased sense of self, even to the point of complete ego death. In some Eastern practices, decreasing sense of self is an explicit goal of meditation (though I see this de-emphasized in adaptations in the individualistic West). Christian meditative practices do tend to have some giving up of the self, though usually the goal is to supplant yourself with god--but I can see this opening people up to beliefs that don't oppose their religion (i.e. the coyotes/polar bears example). Diet, exercise, etc. might help because they shift the balance of fight/flight hormones: it seems apparent that the same hormones that regulate fear of harm to the physical self would regulate fear of harm to the psychological self.
Showing my hand here: there's a number of points here where I'm well outside my own areas of confidence: i.e. on the hormone idea I'm maybe at 20% confidence. I think I can develop a fairly confident position based on a bunch of weak pieces of evidence (i.e. 10 studies with P=0.7 results actually result in a P=0.03 confidence) but it would be easy to change my mind with a single stronger piece of evidence.
How does this relate to values people commit to regardless of the consequences? Consider:
1. A man never backs down from a fight.
2. [Receives vicious beating]
3. A man never backs down from a fight.
I suppose this turns largely on the question of whether "Do X" can be a trapped prior. Since OCD is a thing, probably?
"But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it."
This part worries me a little, specifically that first sentence and the notion that there is some fundamental underlying reality that we can access and know. Inherently the bulk of political disagreement tends to revolve around things that the "scientifically correct"* answer is "we don't know", as opposed to things like "There is a polar bear visible from an LA freeway." The bear thing is a lot easier to develop evidence towards proving than say whether or not gay marriage is a better policy than civil union laws or something. The latter probably requires some robust experimentation over a long period of time to judge properly, where as the former has evidence about that is reasonably easy to get.
Yet people often treat these questions as binary True/False statements instead of Maybe? My suspicion is that this is because for many things an answer of Maybe implies a course of action similar to False: do nothing right now. This stands in direct opposition to those who believe the answer is True, or at least want people to do the things True implies should be done. So you are a badwrong climate denier if you are skeptical.
On the other hand, Maybe looks dangerously close to True from the standpoint of some who believe False. You might be willing to do a little bit of something that shouldn't be done, after all. Give an inch, and those bastards on the other side will take a mile. So you are a badwrong libertarian anarchist if you think there might be some virtue in taking a look at police accountability.
To me, this suggests a problem with politics in general: no one pays the price for their beliefs, so extreme beliefs are optimal. We may dress them up as scientific or evidence based, but most evidence actually points to "We don't know" or "Don't do anything, ever", so it becomes a big game of rationalization, not rational decision making. An external, testable reality has just about nothing to do with it.
*needs bigger scare quotes perhaps
I wonder if the processes you describe are related to the phenomenon whereby a trauma is made worse by how the people around them view the traumatic event. For instance, I recall reading that rape victims who are surrounded by people who consider rape as the worst possible thing that can happen to someone, generally fair much worse than those who aren't in such an environment, ie. people treat their trauma like other terrible circumstances that befall people every day.
Theories I've read suggest that in the former case, the trauma becomes a more central part of their identity, and so they cling to it in some way, but I never quite understood the mechanism behind how this works.
While you were describing how the brain attenuates the sensory input of the traumatic event, and how this just reinforces your trapped prior, this seemed kinda-sorta similar to the trauma scenario above: maybe you don't remember all of the details of your trauma with perfcet clarity, but you have a context (the people around you) constant reinforcing the view that your trauma was the worst possible thing that could happen to a person, and so that heightens your fear of it and traps you in this prior that you are a victim of horrendous abuse that will forever haunt you.
Couple that with the facts that memories are at least partly reconstructed (presumably priors factor into this as well), and I can see how the environment in which a victim is immersed could quite readily affect their recovery and how they cognitively engage with the world.
I nominate a fourth type of bias as equally significant to the three Scott highlights (cognitive, emotional, self-interested). Social bias warrants its own category, I think, because it is so much more than mere external context, -- since society plays such a reifying role in our perception of ourselves, and vice versa, making it a literal extension of ourselves -- and its gravitational power is simply enormous. No doubt one could use the trapped priors model to analyze societies (and subsets) as well as individuals.
I plead guilty of course to all of the above -- but I'm not 100% certain it's pathological. I admit I have a bad bitchy crackers problem, say, watching films of Hitler patting a dog (in fact, every fucking thing he does STILL annoys me!). And I would think we'd have to gear up into some new quantum-leap kind of stage of human development before we'll be collectively ready for a fully rationalist approach where we'll effectively view someone like Hitler without emotional bias during the dog-patting, and simultaneously recognize that he indeed was one of those rare polar bears that shouldn't be denied or ignored. As emotionally painful as it was, it seems relatively healthy that society (at least western, writ large) developed a severe enough trapped prior, lasting more than 50 years, that our "irrational" fear of brutal populist authoritarians actually helped keep them at bay for that amount of time. (Which brings up the question of time passage, at least as a factor when it comes to releasing societal trapped priors.)
"but God coming down from the heavens and telling you anything probably makes apocalypse cultism more probable, not less.)"
There is actually a Talmudic story where God speaks from the heavens telling the majority party of the Rabbi that they are wrong, and their response is "Butt out." (It is not in Heaven. You have told us to follow the majority, so that's what we are doing.)
I believe the source for the evidence that more scientifically sophisticated people are more likely to agree with their side is Dan Kahane, although someone else may have produced similar results. His view is that it is rational behavior. Whether you believe in evolution or catastrophic anthropogenic global warming has almost no effect on the world, since you are only one person. But it can have a large effect on how well you get along with the people around you who matter to you. So it is rational to talk yourself into the belief those people hold even if it is wrong. The more sophisticated you are, the better you are at doing so. This is a little like your rich man/taxes example, but not quite.
For some reason, I can no longer reply to anything except for the top level where I am starting a new thread. I assume this is some sort of bug and may be affecting others.
To me, this sounds very much like a more rigorous exploration of Kolakowski's Law of Infinite Cornucopia "…. for any given doctrine one wants to believe, there is never a shortage of arguments by which one can support it.
A historian’s application of this law might be that a plausible cause can be found for any given historical development. A biblical theologian’s application of this law might be that for any doctrine one wants to believe, there is never a shortage of biblical evidence to support it. "
>>"Along with the cognitive and emotional sources of bias, there's a third source: self-serving bias. People are more likely to believe ideas that would benefit them if true; for example, rich people are more likely to believe low taxes on the rich would help the economy.... I don't consider the idea of bias as trapped priors to account for this third type of bias at all; it might relate in some way that I don't understand, or it may happen through a totally different process."
Maybe I'm misunderstanding, but I'd see "self-serving bias" as following from the model in an almost formal sense, once we add the stipulation that the experience you're integrating with your priors is always *your* experience.
Insofar as, e.g., low taxes are in your (narrowly construed) self-interest, then every time there's a tax cut your direct experience will be of a positive outcome. Maybe you also see some studies or hear some anecdotes about the tax cut being bad for other people, but these seem dubious. Your prior is that tax cuts help people, so if these other people aren't being helped, isn't that more likely to be for some exogenous unrelated reason?
Of course, if you encounter enough evidence of the latter kind, you might eventually come around to the view that your case is exceptional and the optimal policy *for you* isn't the one that's best for society on net.
The point is just that your own immediate experience of [Policy X --> Positive Outcome] is always going to introduce a certain initial weight drawing you away from reflective equilibrium. Which is precisely the metaphor underlying "bias" in its original meaning.
Interesting article for me to stumble on today, as I've been thinking a lot about this dichotomy recently in my personal life and in observing my friends' problems.
Imagine the perfect narcissist. This person has no awareness of his surroundings - he has substituted any real observations for his own internal world. He has no awareness of the emotional states of others, obviously. He does not even have any awareness of his own emotional state.
Such a person could not function in the world. People with narcissistic personality disorder have to be getting in some data - in fact, a lot of the symptoms like rage wouldn't be possible if they weren't. These symptoms occur when the real data contradicts the narcissist's narrative.
Without an external data feed, you absolutely cannot grow as a person. And the process of taking external data, which doesn't give a crap about you, and bending it to your own context and your own needs, is a little selfish.
If you didn't do this, there'd be no difference between you and an insect. Stimulus->Response, with absolutely no middleman. But I'd argue there's a scary other end to that spectrum where your internal labyrinth entirely overwhelms experiential data, and that you can find most mental illness over there.
As a culture, I think we've overvalued our internal world. And maybe the solution to it is to mindfully increase our incoming data bandwidth - make a conscious effort, whenever you think of it, to fight the urge to retreat into your own mind and instead focus on sensation directly. Intentionally give more weight to the sensory over the analytical, the gestalt intuition over the carefully though-out, and the immediate over the future or past.
In other words, stop being rationalist, for a little while each day, and just do some things having no idea whether or not they're good ideas.
Is this a good idea? I don't know.
"seems to create permanent weird false beliefs for some reason"
Maybe there are multiple layers of "trapped priors". As we grow up we develop some base trapped priors that help us stay relatively normal (using the assumptions of the people around us). While psychedelics can help break higher level trapped priors, they also might break some of the lower level trapped priors, allowing the individuals to explore ideas further outside the norm. This can lead them to eventually redeveloping trapped priors that are far outside the norm, making them "weird"
> Although I don't have any formal evidence for this, I suspect that these are honest beliefs; the rich people aren't just pretending to believe that in order to trick you into voting for it. I don't consider the idea of bias as trapped priors to account for this third type of bias at all; it might relate in some way that I don't understand, or it may happen through a totally different process.
Seems like it's probably the same mechanism. You flinch from things that can hurt you. That can be emotional pain (like traumatic experiences), but it can also be expected future pain. "This argument will cost me down the line" is a type of pain to your subconscious, and it seems plausible that it'd have similar defenses.
I feel somewhat that the trapped prior here is the assumption that *what people are doing* when they have intercourse with others is gathering information for the purpose of critiquing their own points of view. But surely that is not obvious. My hypothesis in general is that almost all exchange of information between people is in fact for the purposes of persuasion and/or reinforcement of existing points of view. Id est, people quite rarely converse for the purpose of finding out something new, but much more often in order to reinforce something old (or just for social signaling that sorts other people according to their (old) beliefs).
To take an extreme example, the purpose of a wartime Department of Propaganda reading enemy newspapers is *not* to evaluate whether they should actually be an enemy, but to find new ways to defeat them. Even learning about the enemy's strong points -- at which he is good, noble, and true -- only serves as grist for the propaganda mill: knowing how and where he is strong, we can devise better attacks and defenses. Indeed, knowing the enemy is strong might very well *increase* our devotion to our own cause, inasmuch as we realize victory hinges on a fiercer dedication.
I wouldn't say as a rule we are trying to destroy each other, but there is probably a strong ongoing competitive ecology of social (and even personal) memes among us tribal primates, and it seems not unlikely quite a lot of our intercourse with others represents mere "foreign intelligence" and/or "propaganda" work in service of competing memes.
If that's the case, then we would naturally be confounded by the effect of evidence on beliefs, because we would have the mistaken believe that the mental "engine" at work has as its main purpose the accurate weight of evidence.
I thought it was Napoleon who said "Reasonable men have always a tendency to believe that others are like them, in which point of view they are not reasonable."
It's really too bad the "Rationalists" are unwilling to apply the irrational dog-phobia to their own irrational Trump-phobia:
<i>"Less-self-aware patients will find their prior coloring every aspect of their interaction with the dog. Joyfully pouncing over to get a headpat gets interpreted as a vicious lunge; a whine at not being played with gets interpreted as a murderous growl, and so on. This sort of patient will leave the room saying 'the dog came this close to attacking me, I knew all dogs were dangerous!'"</i>
Some are so non-self-aware they will even admit Trump did not say certain words ... yet still their Trump-phobia makes them CERTAIN that Trump means what he did not say. When Trump says "peaceful protest" they hear "violence and riot".
Since reading The Passion of Ayn Rand, by Barbara, the cheated on wife of Nathaniel who was with Ayn, it was clear that even smart folk, maybe especially smart folk, can rationalize whatever untruth they want to believe. Of course, if they hear "violence" when Trump is saying "peaceful protest", it's something different than mere rationalization. This phobia note explains it.
I suspect this is a semi-widely shared anecdote: one effective antidote to political trapped priors seem to be genuine interpersonal connections. Having discussions with friends you otherwise respect can make you interpret their ideas more charitably than coming from a stranger, and raise your threshold for the mistake theory --> conflict theory trigger. Not 100% sure if either part of this fit in the perception or context part of the model.
Deeyah Khan's white supremacy documentary shows this quite well imo, where a group of white supremacists grew found of her who has middle-eastern parents, through personal interactions, to the point where she (accidentally?) deradicalised a few of them. Many such cases in history ofc. I believe this was also Christian Picciolini's major point.
I think that at least to some extent, the problem with generalizing the scary-dog-friendly-dog example to a policy-evaluation case is that frequently, the "evidence" is not pure in the first place. For anyone but an extremely deeply cynophobic person, it would be harder to interpret more (as-per-non-cynophobe) friendly dog behavior as aggressive/scary, however narrow the evidence band.
What I mean by that is that when the absolute majority of people want to find out what "the science" says about X, they're not actually going to go and read papers - they'll read what their trusted journalists and/or bloggers say the science says, which will be partisan/skewed anyway. Even if they find a source that is very committed to not skewing research, reporting will rely on dumbing down the study to some extent, which inevitably introduces the risk of simplifying the conclusions into something they are not saying.
I wonder how effective it'd be to take someone with trapped political priors completely out of their environment -- both geographical and virtual, to get them away from their friend circles and echo chambers. Being transplanted into a different community would not only increase the evidence they're exposed to for alternative views, but there'll also be a social element to consider. Throwaway comments that are completely based on everyone's collective priors (and evidence) will no longer be welcome, and will be challenged.
As someone who grew up in a highly conservative and religious area and then moved away, I can vouch that some entrenched priors were removed, though it took years in some cases. It's obviously a large undertaking and wouldn't happen unless the person involved is wanting a life change, or up for extreme experimentation. And of course, many of these folks with trapped priors won't *want* to be transplanted into a different community. Nevertheless, if those hurdles are overcome, it could accomplish the goal of bringing the trapped priors back from the "threshold"?
I think the "Point of No Return" is relative to not just the "amount of evidence you'll be exposed to" but more importantly the amount of SAFE evidence you're exposed to - an individual "zone of safety" that constrains what defines evidence.
I think if you have someone who knows what a trapped prior is, and recognizes yours, and they're someone you trust, you can actually get out of a trapped prior pretty easily. As you mentioned: slow and gradual exposure in safe doses.
For something like a phobia, this seems relatively easy to provide. I had a friend significantly reduce my phobia of dogs just by teaching me a few things about dog body language using pictures and videos. I'm still not great around them, but it's a lot easier to deal with them when I need to. I think it was maybe an hour of her time, total?
In particular, she mentioned that a lot of "hostile" dog body language in movies is actually a friendly dog visual, and then angry growling is dubbed in - handling an angry dog during filming is much harder than just using a generic growl! She used a few examples and fixed that prior of mine, and now I can at least tell whether a threat is real or just me having a panic attack.
For other domains, like politics, it's probably a lot harder to develop that sort of trust. But just understanding that some people out there have trapped priors should improve your ability to cut short unproductive debates.
Now I understand that I am probably doomed in covid issues. That's interesting that a year ago I thought myself moderate, I warily supported restrictions though was suspicious of lockdowns.
After five months of being locked up in a tiny apartment and then an escape from Chile (where I lived for five years) to Russia (where I was born and lived before emigration) I became doomed exactly in a terms described by Scott. Now when I hear somebody tells something like "Let's wear masks, this prevents more hard measures" I can't help but think that person is either extremely naive or evil. In Chile people were mostly mask-compliant and they quickly got a long lockdown that is still partially in action (and it is summer there). In Russia many people tried their best not to wear masks, (some people were beaten, at least one killed after they insisted, that others should wear masks) - and there was a short lockdown in Moscow, that ended in a month and no lockdowns after that. Last November an official from my city (Saint Petersburg) said "If people do not wear masks they will not stay at home, so no lockdown". And the same trend you can see through all the Europe and America. So I think every person who enthusiastically wears a mask objectively increases the chance of a new strict lockdown.
Is there any way out of this rabbit hole?
I confess I have suffered from trapped priors. What helped me was engaging in an activity that forced me to often and consistently update my beliefs based on empirical data: trading. This somehow loosened up my trapped prior, as if my whole "system" had become more "open". I believe other activities could have a similar effect. Building something, craftmanship, well I guess anything non-theoretical could work. But 100% agree this is a hugely important field to study.
Practical application of Bayesian statistics often involves probabilities so small that the only way to work with them is on a log scale. Enough very weak evidence over time can overcome a badly specified prior that giving this sort of tiny prior probability to the truth. On a log scale, evidence keeps adding up.
"Ironically, my prior on this theory is trapped - everything I read makes me more and more convinced it is true and important. I look forward to getting outside perspectives."
Your deep dives into the ultimate question of perception, cognition, and everything, when considered together as a series, are beginning to look kabbalistic.
I'm not sure how much rationality as a cognitive exercise has to offer for this particular issue. This idea of the Bayesian Brain is really helpful and informative for conceptualizing a lot of things (I think this paper really influenced that idea: https://www.psy.gla.ac.uk/~martinl/Assets/MCMPS/KnillPouget04.pdf). But biology doesn't actually care about truth, and your brain was not evolved to process information and find truth. That is mostly a happy by-product of the fact that a world model is generally more useful for survive if it corresponds with truth, but that might not always be the case. I think expecting everything to be Bayesian just because some things are is in itself a trap.
Particularly for things directly related to survival (food, safety, sex, group membership) it seems likely that your brain just isn't going to be prioritizing objectivity, maybe to the point where objectivity is physically impossible based on whatever biological implementation these have. If this is true then no amount of information can ever change this sort of belief. You would need to tap into whatever incentive structure created it in the first place.
You didn't mention the role of heros, idols, experts, cult leaders, ideologues etc.
In the case of a phobia, it's not relevant, but for ideological fixation, a trusted expert can go a long way.
Most people don't think for themselves. They set a sort of network of trusted experts and then go with the flow.
These respected allies provide ongoing confirmation that the cult is on the right track.
"If these people, whom I trust, all believe X, then I believe X. If you show me evidence to the contrary, it is necessarily false, however convincing it might seem superficially."
If the cult leader sits everyone down and says hey, it was all a lie (with lots of details)... perhaps some people might stick with the lie but there's a much better chance that many cult members will be released.
The problem then becomes networked falsehoods spread by major influencers. Still a big problem, but not the one you're describing.
Is anybody else upset at the likes being gone? Now I can't distinguish the popular comments from the unpopular ones, which hurts my ability to read the thread.
Perhaps a summary is that priors can “update” input signals before those transformed inputs are used to update the prior. Trapped priors occur when the input signals are always transformed such that they subsequently confirm the prior. This can actually be rational behavior if the reward experienced for having the prior confirmed is sufficiently large.
Having trapped priors in your brain sucks but it's still better than having trapped prions in your brain.
Well, this is #549 for “Trapped Priors As A Basic Problem Of Rationality” : What a loaded title to unleash a torrent of comment.
Humans exist in one physical universe but live in the multi-verses of the mind created by the brain with sensory input, connected by neurologic (somatic and autonomic nervous system), and actuated through muscular systems. Every aspect of these systems exhibit a distribution which influences capabilities, behaviors, and abilities etc which vary in time. Terms such a “standard observer”, “neuro-typical”, normal, abnormal, sane, crazy are used to characterize populations. These latter terms are particularly important to Scott’s primary profession.
Scott characterizes the learning process of the brain’s neural network as “the brain combines raw experience (eg sensations, memories) with context (eg priors, expectations, other related sensations and memories) to produce perceptions. You don’t notice this process; you are only able to consciously register the final perception, which feels exactly like raw experience.” For some reason he prefers the term priors to history and perceptions to thoughts.
Now rationality implies logic. Logic and rationality { https://en.wikipedia.org/wiki/Logic_and_rationality }
As the study of arguments that are correct in virtue of their form, logic is of fundamental importance in the study of rationality. The study of rationality in logic is more concerned with epistemic rationality, that is, attaining beliefs in a rational manner, than instrumental rationality.
Economics: Rationality plays a key role in economics and there are several strands to this. Firstly, there is the concept of instrumentality—basically the idea that people and organizations are instrumentally rational—that is, adopt the best actions to achieve their goals. Secondly, there is an axiomatic concept that rationality is a matter of being logically consistent within your preferences and beliefs. Thirdly, people have focused on the accuracy of beliefs and full use of information—in this view, a person who is not rational has beliefs that don't fully use the information they have.
Debates within economic sociology also arise as to whether or not people or organizations are \"really\" rational, as well as whether it makes sense to model them as such in formal models. Some have argued that a kind of bounded rationality makes more sense for such models.
Others think that any kind of rationality along the lines of rational choice theory is a useless concept for understanding human behavior; the term homo economicus (economic man: the imaginary man being assumed in economic models who is logically consistent but amoral) was coined largely in honor of this view. Behavioral economics aims to account for economic actors as they actually are, allowing for psychological biases, rather than assuming idealized instrumental rationality.
Artificial intelligence:” Within artificial intelligence, a rational agent is typically one that maximizes its expected utility, given its current knowledge. Utility is the usefulness of the consequences of its actions. The utility function is arbitrarily defined by the designer, but should be a function of \"performance\", which is the directly measurable consequences, such as winning or losing money. In order to make a safe agent that plays defensively, a nonlinear function of performance is often desired, so that the reward for winning is lower than the punishment for losing. An agent might be rational within its own problem area, but finding the rational decision for arbitrarily complex problems is not practically possible. The rationality of human thought is a key problem in the psychology of reasoning.
As an example, consider the work on autonomous vehicles which use SLAM: Simultaneous localization and mapping.
https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping?wprov=sfti1.
By analogy, humans navigate in their multi-verse for numerous purposes: staying alive, reproduction, among others. Reasoning and rationality likely evolved in humans because in a Bayesian analysis they are traits that lead to positive outcomes such as staying alive, reproduction, among others.
As an example on distribution, human vision has color, focus sharpness, response time, spectral sensitivity vs light intensity etc.
In the United States, about 7 percent of the male population—or about 10.5 million men—and 0.4 percent of the female population either cannot distinguish red from green, or see red and green differently from how others do. More than 95 percent of all variations in human color vision involve the red and green receptors in male eyes. It is very rare for males or females to be “blind” to the blue end of the spectrum.
The 8% of colour blind men can be divided approximately into 1% deuteranopes, 1% protanopes, 1% protanomalous and 5% deuteranomalous. Approximately half of colour blind people will have a mild anomalous deficiency, the other 50% have moderate or severe anomalous conditions. { https://iristech.co/statistics/ } Women tend to have enhanced blue sensitivity compared to men.
Differences in color categorization manifested by males and females { https://www.nature.com/articles/s41599-019-0341-7 }
Everyone’s multi-verse is potentially shaded differently as is their history
This is certainly consistent with my work as a former military physician on the problem of PTSD from a neurologic (versus cognitive/psychiatric) perspective. Generally the best results when people heal are from eliminating and resolving "trapped priors", which I have seen most commonly (although not always) related to unprocessed shock-level events. I don't find therapeutic value in treating the behavior patterns or trapped priors as independent entities from the experience ... it simply reinforces the avoidance which is a feature of the trapped prior itself.
I have seen this resolution happen routinely by the individual coming into the memory from the outside (i.e. what would you say to encourage that person). This creates a positive association, instead of the reinforced negative of the dog perception example. This could be likened to putting lotion on a burn and has more quality control over the experience ... it's still a burn, but the experience was fundamentally soothing and not irritating, so the relationship with the burn changes (to one of attraction to soothing/love that heals instead of avoidance of pain).
Since most hyperpriors are established by especially important/intense or repeated events, things that stabilize the brainstem function and signal survival are usually helpful - basic breathing and nursing movements provide a bottom-up sensory stimulus of safety that can be accompanied by a top-down interpretation to build a stronger experience ... but it still has to be connected back to the point of unprocessed experience or you will be fighting a long time!
The problem with attempting to overcome priors without resolving the causative events (i.e. reason or stimulate your way out of them without interacting with the primary event) is that you still have unresolved/unprocessed sensory data that will continue to inductively reinforce the prior - strong, unprocessed sensory experiences still hurt (i.e. unprocessed) and serve like an infection which will continue to result in future sensory suppression (which perpetuates the problem) until resolved.
The other challenge is that these are short-term protective (against pain), and so you are fighting against a survival/protective mechanism - that some evidence suggest may relate to physiologic vasoconstriction of focal areas of the body and brain, implying potential therapeutic value in understanding circulatory mechanics.
But on the survival side, as Vincent Filleti (ACES work) says about maladaptive behaviors (substance use, etc ...), "you are dealing with people's solutions, not their problems". We view them as problems, but they view them as physiologic solutions for pain, and that has to be understood in terms of why these are so hard to simply eliminate with a snap of the fingers!
Thanks for the great summary and post! Very stimulating and consistent with my "perception and experience thus far" of the world :) Biases successfully reinforced. Great day!
> The basic idea of a trapped prior is purely epistemic. It can happen (in theory) even in someone who doesn't feel emotions at all. If you gather sufficient evidence that there are no polar bears near you, and your algorithm for combining prior with new experience is just a little off, then you can end up rejecting all apparent evidence of polar bears as fake, and trapping your anti-polar-bear prior. This happens without any emotional component.
I can see that in theory this could happen without emotions, but I’m wondering how frequently we see practical examples (in humans) where the trapped prior isn’t tied to an emotional response.
It seems the cases shown here include:
1. Clearly emotionally-charged subjects in politics, where it’s pretty easy to see the “disgust” / “fear” response flooding the perception and preventing a correct update of priors.
2. Potentially less-clearly emotional cases like tribal / religious identity (cool-headed analysis of political questions that nonetheless fall prey to bias? Religious cults?). Do we want to classify these as not emotional? I could see a case either way. Tribal identity seems like a pretty deeply-rooted part of how the human brain processes social information/situations. And the outgroup responses seem to tie in to emotions at a pretty low level. But I could see an argument that this is distinct machinery from lizard-brain fear/disgust responses.
3. Clear pure-epistemic cases, where there is no emotional flooding blocking the updating of priors. I’m unclear if this set is empty. Is flat-earth-ism one of these? Someone gets infected with a meme that injects the belief “The conspiracy is going to try to convince you that I’m wrong with scientific-sounding facts. If they sound science-y and come from people in lab coats that will be proof that this belief is correct”. Now without any emotions, the update mechanism is subverted. Perhaps we put the cognitive bias of attending more to facts that agree with our previous beliefs here?
> But the context is a very strong prior that dogs are terrifying. If the prior is strong enough, it overwhelms the real experience.
While this is described as the “pure cognitive / no emotion” case, this sounds more like 1, strong emotional flooding. Even without lowering the bandwidth of the present-experience through the trauma response, if the fear association is strong enough it floods/drowns out the “objective” / external experience.
Maybe 3. is actually a different mechanism; 1/2 could be purely explained by emotional flooding causing the subjective experience to be negative, therefore priors aren’t updated. 3. could have something to do with the non-Bayesian nature of the human updating machinery that Scott mentioned in the previous article:
> it can't possibly be this simple, maybe "bad" and "not bad" are binary states and once your brain decides something is bad it fails to process that it was slightly less bad than expected
If “binary” seems too simplistic, it could be some sort of quantization or even just a highly non-linear scale where input of -0.9 is scaled to -0.99999 which would require more than a lifetime of updates to shift meaningfully.
In summary I think this is a useful concept, but there could be two fairly distinct mechanisms at play here that are both causing the overall family of “trapped prior” update failures. In particular it’s not obvious to me that the rationalist experiencing an emotional flood should respond in the same way as a purely epistemic case where cognitive biases are preventing updates from happening properly.
Or as we used to say, One person's modus ponens is another's modus tollens. (Didn't need any of that fancy real-number probability stuff, and we liked it.)
“I'm not sure what level of evidence could possibly convince them.” reminded me of the “arguments” trying to disprove something by “if this were true, some proof would have been found somewhere” but missing the point that for the proponents, the proofs _had been_ found. It’s just the opponents are ignoring the “proofs” as obviously nonsense/crackpot/irrelevant/…
This is exactly what is happening with people who are excessively fearful of COVID (ie, believe that COVID is worse than a once-per-decade flu):
- self-serving bias: I can work remotely and I am saving more money that ever. My life is calmer. Therefore, the economic damage to poorer people is similar, and I don't have to expend too much mental energy drilling down into the question.
- "more scientifically literate people are more likely to have partisan positions on science". In this case (as in the case of eugenics in early 1900s), they "follow the science" without thinking to critically.
"The other promising source of hope is psychedelics. These probably decrease the relative weight given to priors by agonizing 5-HT2A receptors. I used to be confused about why this effect of psychedelics could produce lasting change (permanently treat trauma, help people come to realizations that they agreed with even when psychedelics wore off). I now think this is because they can loosen a trapped prior, causing it to become untrapped, and causing the evidence that you've been building up over however many years to suddenly register and to update it all at once (this might be that “simulated annealing” thing everyone keeps talking about. I can't unreservedly recommend this as a pro-rationality intervention, because it also seems to create permanent weird false beliefs for some reason, but I think it's a foundation that someone might be able to build upon."
If I'm understanding this correctly, I don't think the "for some reason" part is actually mysterious - if a drug makes you more heavily weight actual experiences over priors, and also gives you weird actual experiences uncorrelated with reality, then it makes sense that you'll both be more likely to update on the true evidence you've been collecting and ignoring in your everyday life AND that you'll also be more likely to update on the false evidence the drugs are feeding you.
More generally it seems like this post takes the implicit view that generally people will tend to have overly strong priors and can improve their reasoning ability by weakening them; this may well be true at the population level, but I'm pretty sure it's also possible for one's priors to be too weak. (Which may look like e.g. taking supernatural explanations too seriously when you did not already believe them. (As opposed to a trapped prior in favor of a supernatural explanation, which is common too.))
Maybe this is why there seems to be a rationalists-->[people who are into woo and don't seem to make a ton of sense or be especially good at reasoning] pipeline? For a while weakening your priors has good effects on one's reasoning ability compared to baseline, and then as some point it has bad effects instead?
This is plausible, but let me give a somewhat different theory (as always, the truth may well involve both).
Let's consider the standard model Level-Public explanation of our times:
person did <X> because of <reason couched in terms of ideology>
The salient feature here is the *ideology*. 200 years ago it was Christianity, 50 years ago it was Communism, today it's Woke. The point, in each case, is not that there's a basic moral intuition
- "do unto others as would be done unto you"
- "pay workers a fair wage"
- "treat women like men"
it's that there is a massive collection of deductions as to how the world should be, excuses for why the world does behave the way the model says it does, and a closed off worldview such that the ideological worldview is always able to triumph over "truth", regardless of how truth is demonstrated.
OK, now let's consider a level-N+1 explanation of the prevalence of ideology.
Consider these two statements:
- "the Nazis executed a very clever tactic during the invasion of France whereby they deliberately bombed towns and villages of no military value so as to create streams of refugees that clogged the roads and blocked Allied transport to the front"
- "Ivanka Trump is substantially above average in physical attractiveness"
My experience is that most people (at least, let's say, Democrats for the second) have a real problem with these statements. Their minds rebel at the concept of something being "good" along one axis and simultaneously "bad" along an orthogonal axis, and most people prefer to resolve this by simply asserting that there is no such tension -- the Nazis were not clever, they were simply evil; Ivanka is not pretty, she's also just evil.
Think I'm being ridiculous? Look at the response to Karlheinz Stockhausen's 9/11 comments, eg
https://nymag.com/news/9-11/10th-anniversary/karlheinz-stockhausen/
This n+1 level is a superset of the public level, because it obviously explains the public level ("truth" or <some argument> is good along one axis, but bad [disagrees with the ideology] along an orthogonal axis; let's resolve the tension by picking ideology over truth/<good argument>)
But this explanation is more powerful because it covers cases of tension (like the Nazi case, as opposed to the Ivanka case) that are not objects of active ideological battle right now - few are generating their opinions about Nazis by considering "what have my tribal leaders said about this, what have the tribal opponents said, let's synthesize a response that maximizes the first and minimizes the second")
But we can, I think, do even better. What's we've concluded so far is that most people can't sustain a multi-dimensional "stance" towards something; they feel they have to collapse all the dimensions down to a single-dimensional "good vs bad" continuum (and mostly not even a continuum, two clusters, one of good, one of bad). So why should that be?
I suspect the underlying driving factor is some sort of tribalism inherited from the very earliest social species days. Things were judged in terms of "good for me/the tribe" (and usually, like GM and the US, those were synonymous) vs "bad for me/the tribe". The subtlest this got was maybe something like "Yes, they are our enemies and we want to kill their men. But while we're there we might as well rape their women, always good to have a backup copy of the genes floating around."
This gets us to essentially the same place as Scott, but the mechanism (most minds collapse all value judgements to a single dimension) and the reason for that (ancient tribalism) are substantially different from Scott's explanation (their Bayesian machinery has broken down).
I think my explanation is more powerful than Scott's (though his mechanism may be a part of how my larger mechanism works) because Scott's machinery doesn't really have a story for the sort of example I gave for clever Nazi's, cases where people don't want to admit a good vs bad tension, but this is not really part of living in a bubble/locking one's priors to absolute truth.
(And such cases are common! There are similar very clever financial manipulations [that hurt a lot of innocent people], or beautiful art works by horrible people, or very clever design elements in <chip/operating system from Apple/Google/MS/Intel/Facebook/whichever tech giant you choose to hate>, or elegance in the design of a fusion weapon.)
Step 1: first we reinvent Freud's theory of repression:
> Van der Bergh et al suggest that when experience is too intolerable, your brain will decrease bandwidth on the "raw experience" channel to protect you from the traumatic emotions.
Step 2: then we pull from one of Freud's favorite influences, Spinoza, to reinvent his "associative" theory of emotions:
> you're in an abusive or otherwise terrible relationship. Your partner has given you ample reason to hate them. But now you don't just hate them when they abuse you. Now even something as seemingly innocent as seeing them eating crackers makes you actively angry.
Step 3: then we reinvent Zizek's theory of ideology, but without the parts that specifically focus on how it's specifically *symbols* that people tie their deep priors to (i.e., you can tell a Republican about a program that Democrats would support, and they will be in support of it so long as you don't tell them that it's a *Democrat* program).
> zealots' priors determine what information they pay attention to, then distorts their judgment of that information.
There's also a subtle normative/ethical stance in Scott's post, by implying that some political stances are more "detached" from "reality" than others, i.e. that in an entirely symbolic game, some people are making "wrong" or "bad" decisions which we need to "fix".
Toss in some REBUS, and we're back in MK-ULTRA territory, trying to use psychopharmacology to "fix" people's beliefs, a program that worked "very well":
> Its goal would be a relatively tractable way to induce a low-prior state with minimal risk of psychosis or permanent fixation of weird beliefs, and then to encourage people to enter that state before reasoning in domains where they are likely to be heavily biased.
Finally this:
> Tentatively they’re probably not too closely related, since very neurotic people can sometimes reason very clearly and vice versa, but I don't think we yet have a good understanding of why this should be.
My stance is that they're absolutely related. Think of a "trapped prior" (i.e. a traumatic kernel, in Freud's language) as a "black hole" that "warps mental space-time" for the individual. When you're traveling around (thinking) far from the black hole, then the thoughts "make sense", but the closer you get to the black hole, the more things get distorted, irrational-seeming.
The problem is, as Lacan says:
> there is no position of 'mental health' which could be called 'normal'. The normal structure, in the sense of that which is found in the statistical majority of the population, is neurosis, and 'mental health' is an illusory ideal of wholeness which can never be attained...
So space-time will form black holes, the only question is where are they located. Perhaps you want people's mental space-time to warp in more familiar places ("my bias is actually the normal"), and want to establish an ethical program to ensure that more people are "like me".
This wouldn't be the first time that institutions or groups have embarked on such a program, but I urge you to think through the ethical implications, if "eliminating bias" no longer has a "privileged" position with respect to "reality". One thread to pull on: what happens to the ecosystem of thought and communication if we successfully eradicate "diversity of bias"? And what will the material consequences be?
I'd like to take a different lesson from the psychedelics research. They focus a lot on 'set' and 'setting'. This is the idea that it matters how you enter into and in what context you experience a trip. They focus a lot on this, and I've heard some insist that it's entirely the difference between a good and a bad trip.
I've never used them myself, but I hear a lot of descriptions of psychedelics as effectively eliminating the 'prior' side of the models above, leaving only sensory experience as an input.
There's a place we can build this into Scott's model. He brought up the idea of weighting priors/sensory evidence, but never asked the question, "What's the weighting function?" Yet from his desensitization example, as well as the discussion about emotion-motivated setting of priors, we can make the following guesses:
- Priors are weighted by the emotional context under which they were established
- Sensory evidence is weighted by the set/setting under which it is experienced
This is why the gradual approach of pictures of puppies works well, but the locked room doesn't. It's not that priors can't be reset, it's that the sensory evidence that should reset them only gets sufficient weight when amplified by the correct modulator (set and setting).
Andrew Huberman often refers to the cognitive regulation of emotion through eye movement - there is evidence that a subject recounting a traumatic event while moving the eyes laterally extinguishes the fear or distress. This kind of study: https://www.jneurosci.org/content/38/40/8694
> this might be that “simulated annealing” thing everyone keeps talking about
Umm, who's talking about this? I feel deeply out of the loop, but google is just returning results from computer science. Is there a fad among rationalists of trying to overcome biases by believing obviously-false claims to see if they turn out to be true, and they're calling it "simulated annealing," and I haven't heard about it?
Some thoughts based on the observation that computational power (both operations and data storage capacity) isn't free.
*****
Speculation: the writeback of the updated prior is a nontrivial portion of the total energy cost of a Bayesian update in humans.
If so, then one obvious optimization from an overall-energy-cost point of view is to only do a writeback if the prior has changed by a non-negligible amount.
...and then you end up in the sorts of trapped priors that you describe.
*****
One other related observation: you end up with exactly the same sort of thing with deterministic rounding modes and restricted-and-finite bitwidths for the prior, such as you would end up with if you were trying to minimize storage requirements.
(If the rounding was _non_deterministic and the bitwidth small then you could end up in a funny scenario where small amounts of progress generally wouldn't happen - you'd see a nontrivial amount of progress or none. I don't know enough to know if this sort of thing happens.)
I love this piece. An assumption of Bayes Rule is that priors do not influence new evidence, but obviously they do. If magnitude but not direction of evidence is altered, then we still converge on truth. If direction is altered, we have "trapped priors".
Maybe we can explain the trap if there is a prior on the expected physiological reaction that constantly underestimates the physiological reaction because it is tainted by some rational expectation. My rational brain tells me that being with a dog will be ok. But my physiological reaction turns out to be quite bad for no rational reason. That might reinforce the physiological reaction (the reaction was actually bad and much worse than I expected overall) and it may keep the rational expectation unchanged because come on, puppies. No clue whether rational vs physiological priors are a real thing but it seems hard to escape.
Yet, as Stephen Senn persuasively argues, "the placebo effect is largely a myth." Perhaps a more profitable approach would be to figure out how so many people avoid falling for illusions.
I used paint's color picker to check if the raw pixel data at the chess pieces in both the images are grey. I tried to avoid the smoke that was hard. https://imgur.com/avQC6Kx I think the white pieces are actually grey and look whiter than it actually is. But the black pieces are actually black not grey. I think it would be better if you could post a 3rd image without the smoke.
I still can't get the image of some monks stuck in a precarious position every time I read 'trapped priors'.
I think perhaps "bitch eating crackers" is a poor example, because other people eating crackers is mildy annoying. If a stranger were eating crackers opposite me on the train, I might be irritated by their lack of consideration. If a friend is eating crackers, I'm less likely to be bothered. In part, I think that's because I *could* ask my friend to stop if it was bothering me too much (whereas I would feel uncomfortable making that request of a stranger), and the mere fact that I could make the request puts the cracker noise within my domain of influence, rather than being something externally imposed.
With the toxic relationship, the problem is (I speculate) that the guy fears that if he asks his partner to stop eating crackers, that would trigger a massive row. In turn, the mere fear of that row, without his partner actually doing anything, makes him furious, because what sort of relationship is it where you can't even make a such a simple request? That might all be paranoia, in that the partner would actually happily refrain from eating crackers if she knew it was annoying, but it *might* be completely accurate. The partner really might understand a request to desist from cracker eating as hostile and might react negatively to that. If so, there really is a problem in the relationship. In that case the guy's problem isn't epistemic but pragmatic: he correctly identifies a problem in the relationship, but is unable to identify a solution and (perhaps) reacts in a way which only exacerbates the situation.
I don't have time to read all the comments, so I assume someone may have addressed this. But this isn't an issue with Bayesian processing in my view. If we assign a 0% probability to an outcome in our prior, then Bayesian updating will produce "trapped priors". This just highlights the fact that regardless of the data observed, the observer must be open minded about change.
In my experience, Socratic questions opens priors. But it is exhausting to pursue and takes forever.
> (this might be that “simulated annealing” ...
closing parenthesis is missing
No you don't. You have made that very explicit in our one on one conversation. You want to want to get outside your perspective, but you accuse me of witchcraft on the grounds that I have a historical track record of getting people to change their perspectives.
While this discussion is about phobias, traumas and negative priors, I wonder if it also works for some positive priors / emotional orientations toward things? ie whether having a massive crush on someone means that any time you interact with them, you get a hit of good feelings (or "they make you feel really good" as you may interpret it), which strengthens your prior that they're genuinely wonderful, which means next time you see them they make you feel great again... people do seem to get temporarily trapped here, but it doesn't entrench the way phobias do. Maybe because our brains are a lot more reactive to single bad things than to single good things?
A recent paper tried to replicate the Fernbach et al findings referred to above, and found that they didn't replicate.
"Asking People to Explain Complex Policies Does Not Increase Political Moderation: Three Preregistered Failures to Closely Replicate Fernbach, Rogers, Fox, and Sloman’s (2013) Findings
Fernbach et al. (2013) found that political extremism and partisan in-group favoritism can be reduced by asking people to provide mechanistic explanations for complex policies, thus making their lack of procedural-policy knowledge salient. Given the practical importance of these findings, we conducted two preregistered close replications of Fernbach et al.’s Experiment 2 (Replication 1a: N = 306; Replication 1b: N = 405) and preregistered close and conceptual replications of Fernbach et al.’s Experiment 3 (Replication 2: N = 343). None of the key effects were statistically significant, and only one survived a small-telescopes analysis. Although participants reported less policy understanding after providing mechanistic policy explanations, policy-position extremity and in-group favoritism were unaffected. That said, well-established findings that providing justifications for prior beliefs strengthens those beliefs, and well-established findings of in-group favoritism, were replicated. These findings suggest that providing mechanistic explanations increases people’s recognition of their ignorance but is unlikely to increase their political moderation, at least under these conditions."
https://journals.sagepub.com/doi/abs/10.1177/0956797620972367
I haven't looked into this literature myself, however.
There's an interesting discussion on this issue in E. T. Jaynes' "Probability Theory: The Logic of Science". Section titled "Converging and Diverging Views" on p113. If you like Bayesian probability theory and rationality, you really will enjoy this book.
I suspect our mind pulls a simple but effective trick on us. When faced with new sensations that run counter to our strong prior, we actively remember past sensations (scary dog memory, all of my old arguments against capital punishment). But we _falsely_ interpret this rehashing of old facts as new input. Repetition of already known facts is an effective tool of persuasion even if it should have zero value from Bayesian point of view. And we do it to ourselves. So our strong prior is in effect updated with weak evidence against and strong (echo) evidence for.
You might see this play out when reading an oped you disagree with. Rather than only interpreting the new input and judging it as unconvincing, you’re probably pulling out some of your favorite counter-arguments and (if you’re anything like me) repeating them loudly to anyone unfortunate enough to be within earshot. Being already known, they should not update your prior. But I suspect they actually do.
Something that doesn't make sense to me about this model: If VdB et al suggest that experience is decreased to protect from negative emotions, that suggests that emotion belongs to 'sensation' or the empirical input of experience. But surely, 'scary' refers to the emotion of fear ('dangerous' would be the emotionless empirical equivalent), yet here the model depends on a prior of a scary, not a dangerous dog, suggesting emotion belongs instead to the predictive 'prior' input to experience.
Also, it just seems obvious that people can and do deliberately eat crackers in a bitchy way.
Look forward to advertising and PR agencies getting their hands on that research that shows a reliable method for releasing trapped priors. Every aspiring cult leader will want to make use of that data.
"The biggest problem I used to have is people used to have intractable biases against buying the snake oil I was trying to sell them."
The fundamental error of this approach is the need to explain biases in the first place. Unbiased ness is suboptimal in finite sample inference. Before you start looking for explanations , first you need to make sure you have the proper model of optimal cognition.
I would recommend using a dial in the center of the gray box to represent weighting. That way you can still use your restricted pipe metaphor while also capturing what the color output would be normally.