I’ve decided wrtie a post on the topic of self-confidence for three reasons, all having something to do with religion: First, there are the rather extraordinary claims made by New Agey types about the supposed power of belief to change the world in any way you please. Second, self-confidence is a frequently cited (by no less a philosopher than William James) as a case of belief that’s okay to have in spite of lack of evidence. Finally, there’s a lot to be said for the thought that most people have way too much self-confidence, something that’s of interest to anyone who cares about having rational beliefs.

If you’re like most people who read blogs like mine–religious skeptics who do a fair amount of reading and thinking about religion–you probably have the easiest time deciding what to think about the third point. You’ve almost certainly encountere people who can be ridiculously confident about their beliefs even when wrong. You’ve probably heard about the psychological research done on this subject, which has produced such nice gems as the fact that most people rate themselves “above average” in damn near everything. If you’ve read conservative commentators railing against the self-esteem movement, you may have decided that’s the one issue where you’ll throw in your lot with theirs.

Once you’ve got that, it can feel like you’ve got an easy answer on the first two points: excessive self-confidence is bad, we should be trying to bring our beliefs about ourselves in line with reality, so self-confidence isn’t an example of a belief that’s okay to have in the face of self-confidence. As for the supposed power of belief, attributed to everyone from western faith-healers to eastern coal-walking monks, there’s no evidence for the extraordianry things attributed to it, and it should be seen as a con used by charlatans to keep their followers from thinking rationally about their claims.

The trouble with that easy answer is this: some of the less-extreme claims made by preachers of the power of belief are supported by the evidence. One especially tidy experiment found that when Asian women are given a standardized test, they do better if they are first reminded that they are Asian, worse if they are first reminded that they are women. And while the effects found in standardized tests are relatively small, the effects on one’s success in social situations can be enormous–anyone who doubts that should spend awhile studying Tucker Max’s website.

I’m somewhat sympathetic to the New Age types who see these facts and conclude there might be something to the idea that beliefs can cure the sick and protect you from burns. But unfortunately, there’s no evidence that mere belief is enough to get you a perfect score on a test, much less to do anything supernatural. More decisively, there are natural explanations for why beliefs have what power they do have.

The explanation for the social benefits of self-confidence is easy: the human brain is constantly drawing subconscious conclusions based on usually-reliable signals that we’d be more wary of if we realized what we were doing. For example, the free market ensures that price normally correlates with desirability, so price is a signal for desirability, but this means that in some situations, businesses have found that they can increase sales by raising the price of their goods. Similarly, hackers know that if they have a little bit of inside trivial about an organization–like names of employees–they can often use that to talk their way into getting whatever they want because, if you drop a few names, people assume you’re an insider even though no one is stupid enough to consciously think “top-secret information can be given away to anyone who knows the right names.”

What confidence does is act as a signal that we’re used to having things go well for us, used to having people respond to us well. So, when we see a confident person, we automatically assume that they have something going for them–and that will amount to a very big thing going for them.

Why confidence would have anything to do with standardized tests isn’t as clear, but there are a couple plausible evolutionary explanations. It may, for example, have something to do with not wasting effort in tasks you aren’t likely to excell on anyway. Or, it may be that in times past, when members of society who weren’t “supposed to” excell excelled, they tended to get put in their place in a rather nasty way.

But okay, you might say, just knowing that the loopier New Agers are wrong doesn’t tell us what to do with all this. What do we make of James’ argument? Should we accept that it’s sometimes rational to believe without evidence, or when the evidence is ambiguous? The problem with William James style arguments is that they fail to distinguish between practical rationality and having rational beliefs. It’s perfectly coherent to advise someone, as a practical matter, to believe things that are irrational as beliefs. Indeed, such advice sometimes is given (scroll down to #11; HT: Overcoming Bias).

And just as a practical matter, the sort of over-confidence most people have has real drawbacks. It gets in the way of understanding how the wold really is. It can lead you into foolish displays of cockiness. And for most people, all the little lies they tell themselves don’t add up to confident behavior, their behavior, rather, tends to betray their insecurities.

The path that gives you the best of both worlds–the benefits of real self-confidence without the costs of pedestrian self-deception–is the one where you do your best to see the world as it really is, including a realistic understanding of what greater self-confidence can do. This will allow you to have more self-confidence where doing so will be a self-fulfilling prophecy. Also, it’s possible to behave more confidently independent of your beliefs about whether you’ll succeed. Instead of overestimating the chances of success, realize the costs of failure may not be as great as you’re inclined to think. And it’s possible to ask yourself “what would I be doing if I had no self-doubts” and then go do that, in spite of your doubts.