The Art of Doubt

Do you believe in ESP?  Precognition?  Clairvoyance?  Telepathy?  Psychokinesis?  Assuming you don’t (!!), how do you argue against such things?  It’s pretty simple, right: you apply the critical thinking techniques that are essential to science:

  • Is the theory falsifiable?  In other words, what evidence would be required to disprove the theory?  To say that a theory is not falsifiable is to say that the evidence is irrelevant – nothing could shake your belief in it – which, logically, makes the theory pretty meaningless.
  • What does the theory predict, specifically?  A common way to get around falsifiability is to make the theory so vague or broad that it doesn’t have any real content to it.  Good science makes specific predictions (and is useful for it!)
  • Has all the evidence been presented?  You can prove more or less any theory if you throw away the evidence against it!  If you’re trying to disprove something bogus, you may be able to unearth hidden negative cases.
  • The burden of proof lies with the claimant; it is not for you to justify your disbelief.  It’s easy to make up wild theories that are actually quite hard to disprove (aliens live on Mars and hide from us on the far side!); they’re fairly pointless unless some evidence actively supports them.
  • Personal observation and testimonials form inadequate evidence.  People make mistakes and lie, their recollections can be incomplete and highly subjective.  Unfortunately, humans are hard-wired to enjoy learning through stories, and often attach disproportionate credibility to a well-told tale.  Analogies suffer a similar effect; they don’t necessarily prove anything but can be highly persuasive.
  • Is the proponent honest with their evidence?  Are they really trying to find the truth, in an open-minded manner?  One easy way to spot dishonesty is when you notice strong emotional commitment to a theory.
  • Can experiments that support the theory be replicated?  Never mind dishonesty; you also have to be able to rule out coincidences, or other external factors that could provide alternative explanations.
  • Are the arguments in favour of the theory logically sound?

It’s not necessarily a core principle of science, but as a modern practicality, you also have to consider sources of information.  Information is frequently quoted and passed along through whole chains of publications; our information-storing and analysis technologies allow these chains to become longer and more complex than ever before.  Now, using information in this way is a powerful thing, allowing people to stand on the shoulders of others.  But if a theorist wants to be rigorous, they need to track any such information right back to its source: there are often surprises to be found.  The people quoting along the way may well have made mistakes, and even when they didn’t, they may have failed to describe original assumptions and conditions (which may not have been important to them).

Finally, if you actually end up arguing against pseudoscience, you have to be ready to counter some dubious techniques of argument.  One favourite is the ad hominem technique.  The ‘believer’ turns on you personally, making the debate about you: your motives, your stupidity, or whatever.  Always remember: these are definitely not arguments in favour of the theory; if you can spot them, you can reject them as such immediately.  Actually, there are many more techniques, and they’ve been well-studied.

Why am I writing about the paranormal???

Well, I’m not 😉

Old Japanese fortune teller

When writing about software measurement, I mentioned a few of the false arguments pro-metrics fanatics throw around.  Looking back at them, I certainly described these: not accepting the burden of proof, bad analogies and personal testimony, excuses for non-replicability and ad hominem attacks.  I didn’t mention them, but lack of falsifiability, lack of specificity, discarding negative evidence, emotional commitment and unsound logic are all pretty easy to find in the pro-metrics camp too!

But I don’t want to pick on software metrics again just now.  Bogus arguments and pseudo-science, unfortunately, permeate the world of management theory.  It is incredibly hard to find theories of management (where ‘theories’ includes any process, technique or methodology) that are based on strong evidence and scientific reasoning.  For one thing, it’s very hard to do real experiments with management techniques.  Nobody can afford to run a real-life project of substantial size many times over with different techniques, control groups and so on (nor to repeat the results!).  Even if they could, there would be too many uncontrollable variables.  Good management is hard, so there’s a big market for theories as desperate managers look for a silver bullet.

You need to be aware of these critical thinking techniques when examining any new management theory.  I’m not suggesting you shoot down every management technique you come across; that would be too easy, and you’d be so negative that you’d never take on the good ideas that are essential to improvement (there are many).  It may be that an idea is useful to you despite poor supporting arguments; it may be that you can make use of parts of a methodology while discarding others; and it may be that a theory, while not universally true, can be valid under the right circumstances.

These techniques simply help you to think for yourself: to discard the claptrap, to think about what conditions are necessary for an idea to work, to take the good parts of a methodology and discard the bad.  I think I’ll be coming back to this almost as a checklist when I examine some management ideas in the future 🙂

This entry was posted in Management for Geeks. Bookmark the permalink.