Showing posts with label evidence. Show all posts
Showing posts with label evidence. Show all posts

Tuesday, 12 February 2008

Knowledge and pseudo-knowledge

It’s often easy to distinguish prejudice from informed opinion. Prejudice is an opinion ‘without visible means of support’ whilst the holder of an informed opinion can back it with arguments and examples. The holder may be part of a professional community with its own journals, models, prizes, etc.

But is every subject with these features a form of knowledge? It turns out that quite a lot of them are not.

One relatively easy way to determine whether a subject is really knowledge is to ask whether the people practicing it can make valid predictions. By valid I mean that their predictions are correct significantly more than half the time and much more often than the predictions made by non-experts.

[Some subjects, eg theology, media studies, contain many theories that make no predictions. I have discussed three of these separately.]

Pseudo-subjects are rare, possibly unknown, in science and engineering because these subjects have institutionalised methods for testing their theories. They are much commoner in medicine largely because of the power of the placebo effect. For instance, Homeopathy makes predictions about the effects of its treatments. When carefully tested to exclude placebo effects, eg in double-blind clinical trials, these predictions are generally wrong.

They are commoner still in the subjects that study human activity, eg Politics, Economics and the humanities.

In The Black Swan – The Impact of the highly improbable (Allen Lane, 2007) Nassim Taleb looks at the evidence on the accuracy of predictions made in Security analysis, Political science and Economics. In each case he finds there are few studies but that those that exist show the predictive power of these fields to be poor.

Security analysis

Securities are tradeable financial instruments such as shares, options, collateralised loan obligations. The job of the securities analyst is to advise investors which to buy and which to sell.

T Tyszka and P Zielonka (Expert Judgements: Financial analysts versus weather forecasters. J. of Psychology and Financial Markets, 2002, vol 3(3), p 152-160) found that, compared to weather forecasters, security analysts are worse at predicting but have more faith in their predictions. An analysis communicated by Philippe Bouchard showed that predictions in this area are on average no better than assuming this period will be like the last period, ie worthless.

This is despite the analysts’ extensive knowledge of the firms and sectors they study!

Political science

Philip Tetlock (Expert Political Judgement: How good is it? How can we know?, Princeton University Press, 2005) asked about 300 supposed experts to judge the likelihood of certain events within the next five years. He collected c27,000 predictions. He found:

· Experts greatly overestimated their accuracy

· Professors were no better than graduates.

· People with strong reputations were worse predictors than others.

Economics

Taleb found no systematic study of the accuracy of economists’ predictions. Such evidence as exists suggests that they are just slightly better than random. For instance Makriades and Hibon (The M3-Competition Results, Int. J. Forecasting, 2000, vol 16, p 451-476) ran competitions comparing more and less sophisticated forecasting methods. They found that sophisticated methods, like econometrics, were no better than very simple ones.

Common threads

In each case:

· The experts know a lot more facts than the layman.

· They also know more theories about their fields.

· There is little published research about the accuracy of these theories.

· Supposed experts are not familiar with what little there is.

· Predictions cluster – experts copy each other.

· Predictions do not generally cluster around the true value.

These fields may have value as sources of analogies and models. Sometimes even remote analogies and very simple models can be helpful.

However, their predictions are generally worthless.

Friday, 18 January 2008

Why knowledge is NOT “justified true belief”

In Theaetetus, Plato’s Socrates argues that knowledge is “justified true belief”. That is, it is, a belief for which the believer has a justification and that is, in fact, true (see en.wikipedia.org/wiki/Epistemology). This doesn’t really work.

Firstly defining knowledge as belief excludes tacit knowledge, such knowing how to ride a bicycle. My ability to ride a bicycle does not depend upon my having any particular beliefs about riding or, indeed, bicycles. So Socrates’ definition can only apply to explicit knowledge.

To see why it doesn’t even work for explicit knowledge consider the following truth table for beliefs.

1

2

3

4

Is the belief justified?

Y

Y

N

N

Is the belief true?

Y

N

Y

N

Does it meet Socrates’ definition?

Y

N

N

N

Justification: I agree with Socrates in wanting to distinguish knowledge from mere guesswork. To count as knowledge a belief needs to be justified. Suppose you believe that Hungary is an attractive market for your latest product. This is a justified belief if you can produce evidence for it. I would accept market research and the success of similar products as sufficient evidence so if you have that data I’ll accept your belief as justified. I’m prepared to agree that a belief is justified even if the evidence is not conclusive, as it won’t be in this case until you’ve tried to sell it in Hungary.

Truth: The problem comes in distinguishing cases 1 and 2 in the table. Is your belief about the Hungarian market true? The only way that I can know this is to wait for the results of your Hungarian launch. And if you never launch your product in Hungary then neither of us will ever know if your belief was true – and therefore knowledge. This is inconvenient but not absurd.

How about the market research? Is that knowledge, ie is the data you have accurate and appropriate? You believe so and you have reasons for doing so, eg that you used a reputable firm with experience of the Hungarian market. You certainly have a justified belief. But can you know that it’s true? You cannot be certain, so this, also, does not meet Socrates’ definition of knowledge.

Not truth but confidence

The reality here is that beliefs can be held varying degrees of confidence. We can rarely be certain but are often, justifiably, confident. And we determine the proper degree of confidence using just those arguments that constitute the justification for the belief.

So knowledge, I suggest, is those beliefs where the arguments and evidence in favour make them seem more likely than not. Beliefs where the arguments and evidence in favour fall short of that are rumours, opinions or prejudices.

This view has two immediate consequences:

  • In applying knowledge we should consider how sure we are of its truth. We should not claim certainty where the evidence is doubtful.
  • Systems that store knowledge should include assessments of reliability and/or pointers to the sources of the supposed knowledge.

Tuesday, 15 January 2008

TNDM: A predictive model for warfare

The Dupuy Institute's Tactical Numerical Deterministic Model (TNDM) (Economist. Technology Quarterly; 17 Sept 05) is a forecasting system with an excellent track record in forecasting the durations and casualties in armed conflicts. The Institute has a database of raw information on which its analysts perform extensive statistical analysis to identify patterns and trends. TNDM is available commercially (for $93,000 in 2005) and has been bought by both governments and arms suppliers. The Swedish government uses TNDM to propose new kinds of weapons.

The model includes many factors some technical, eg types and characteristics of weapons and armour, some geographical, eg presence of rivers, some tactical, eg disposition of troops, some logistical and even the matter of morale.


TNDM is generally more accurate than other 'war forecasting' systems because:

  • It's based on real data
  • It's model is based rigorous analysis rather than the wishful thinking of arms manufacturers and military organisations.
  • It's model has been repeatedly tested against actual experience.
The success of TNDM is significant because it deals with human behaviour (which is sometimes claimed to be wholly unpredictable) and because it shows the success of empirical, 'scientific' method in an area remote from the physical sciences.

Saturday, 29 December 2007

The starting point

I've been thinking about knowledge and knowledge management intermittently for over ten years. As a computer and management consultant I started from the need to manage information and knowledge in organisations and from Peter Drucker's insight that 'knowledge work' was becoming central to the economy.

I believe there's a disconnect between the two. By knowledge work Drucker meant, roughly, work done by people with formal education at or above degree level. But most of what's written about 'knowledge management' (KM) has nothing to say about the kind of knowledge that is learnt in gaining a degree. I don't say that the KM literature is worthless - only that it's incomplete in an important respect.

My own thinking started from a definition and two propositions.

My definition

Knowledge is assertions that are based on evidence. Assertions don't become knowledge because they are widely believed or found in textbooks (or holy books) or endorsed by authority (whether political, organisational or religious) but because they are supported by logic and observation.

Notice that this is NOT the same as defining knowledge as "justified true belief". Truth is not part of my definition because it cannot be definitively known.

Scientific knowledge is the form of knowledge that most clearly exemplifies my definition and it's reasonable to seek illumination from the philosophy and practice of science. There is, of course, knowledge in other areas, eg history, marketing and business management.

Of course my definition begs the question as to how much evidence is needed to to convert an assertion into knowledge. I confess that I do not (yet) have an answer to this question of which I'm confident. My best current answer is to replace the opinion/knowledge dichotomy with a scale which has prejudice at one end and established knowledge at the other. This would be consistent with Russell's advice to 'give to each proposition that degree of belief justified by the evidence'.

Two propositions

A) Knowledge is most interesting and important where it is general. Thus Newton's laws of motion are more important than the orbital parameters of any one planet. And an effective customer segmentation scheme is more important than any one customer's spending pattern.

B) Much knowledge can be seen as a model of the domain concerned. Newton's laws of motion are a mathematical model and a great deal of science can be seen as models of various kinds.

Models, generally less precise and mathematical, are also common in management science and marketing.

Some knowledge, and this is especially true of tacit knowledge, is probably not models; it's certainly not explicit models. I'll return to this point in subsequent posts.

Implications
The processes of knowledge creation and verification are important parts of knowledge management. It's useful to see them as model building and testing.