• We have updated the guidelines regarding posting political content: please see the stickied thread on Website Issues.

The "Four I Test" (Discursive Article)

Mikefule

Justified & Ancient
Joined
Dec 9, 2009
Messages
1,282
Location
Lincolnshire UK
Part of my job used to be designing training material for a small team investigating household and motor insurance fraud. For those of us who are interested in the sort of Forteana that involves reports of anomalous sightings (UFOs, Bigfoot, Loch Ness Monster (LNM), ghosts etc.) there are some similarities to the thought processes.

I thought I’d share some ideas here for those with a broadly similar approach to Forteana to my own: what I think of as “honestly sceptical” — as distinct from being determined to debunk everything as a hoax or misidentification.

The insurance claims investigator and the honestly sceptical Fortean are both faced with reports that may or may not be true, supported by unusual or unsatisfactory evidence, or sometimes by none. In both cases, they normally have to work to “the balance of probabilities” rather than applying scientific levels of proof or the criminal law standard of “beyond reasonable doubt.”


In the early days of developing training material, I relied on two common maxims that most Forteans will recognise, whether or not they agree with them:
  • “Extraordinary claims require extraordinary evidence.” (Carl Sagan)
  • “Absence of evidence is not evidence of absence.” (Attributed variously)
I soon concluded that these were unreliable.
  • An extraordinary claim requires only sufficient evidence to support it. The standard of evidence does not have to be higher just because something is unusual. In fact, the converse is more nearly true: a claim that is in no sense unusual may often be accepted with minimal evidence or none — unless of course there a specific reason to doubt it.
  • The absence of evidence that would normally be expected to be available is a form of evidence.

Putting these into context:
  • An extraordinary insurance claim may be the theft of a £10,000 diamond ring from a person with low income and with a home and lifestyle consistent with that low income. However, a document showing that the ring was recently inherited, accompanied by a jeweller’s valuation, would be sufficient. That is the same standard of proof that would be required if a wealthy person were claiming for a ring of similar value. However, in both cases, a claim for a £150 ring might be accepted with no evidence other than a consistent description of the item.
  • An extraordinary claim to a Fortean might be a sighting of the LNM. However, 10 seconds of focussed video footage showing clear morphological detail, with background to provide context and scale, and with verifiable provenance and unedited file properties, would be all it would take to persuade us that it was a genuine sighting. That is not actually a lot of evidence.

  • If a customer, however wealthy, claimed for a £10,000 Patek Philippe watch, the absence of a receipt may not be important. However, imagine a customer who could not provide a receipt, or details of where he bought the watch, or a bank statement showing the payment, or the manual, or the certificate of authenticity, or service records, or even a photo of himself wearing it. The insurer would not be entitled to insist on all of these things, but they would be able to argue that he could not provide at least some of these things rendered the claim “not yet valid”.
  • In the case of the LNM, if someone texted me to say that he had been sitting by the loch for 20 minutes watching the monster at close range, but he had no photos or video of it, I would not believe him. However, if he told me he had seen a red kite 10 miles outside its usual range, I would happily take his word for it.
So, having set aside these two maxims as unreliable for analysing potentially fraudulent claims, I devised what I called the “Four I Test.” I asked my colleagues to ask themselves four questions about any claim that they were considering.

In this order: what part of the claim, if any is:
  • Implausible
  • Improbable
  • Inconsistent
  • Impossible

I think this is a potentially useful approach to any anomalous report. Let’s unpack those four words:
  • “Implausible” means that something is not convincing, or does not seem likely. This says more about the person who is unconvinced than it does about the subject they find unconvincing.
For example, a colleague once said to me, “I don’t believe anyone would spend that much on a child’s birthday present. It’s not as if it was a special birthday. I don’t believe them.” I pointed out that this may be how things were in her family, but someone from a different culture, part of the country, or income bracket, might be equally surprised that her own family spent so little on birthday presents.

In the case of Forteana, we are perhaps all guilty of finding things plausible or implausible depending on our own life experiences and preferences. For example, I often go out into the countryside without my smartphone, so perhaps I find it easier to accept that someone who claims to have seen an ABC did not have their phone with them at the time.

  • “Improbable” means that something can happen, but it is very unlikely. The chances of winning the National Lottery are 45,057,074 to 1 if you buy one ticket. It is incredibly improbable that you will win. However, every week or so, someone does.
In the case of an insurance claim, it may seem improbable that a burglary happened at 7:00 in the morning (most are mid afternoon) in the 10 minutes that the householder was down the garden, but it is not impossible.

In the case of Forteana, it may seem improbable that the same person saw an ABC on their first visit to Dartmoor, another on their first visit to the Norfolk Broads, and another on a day trip to the Peak District. However, if you accept in principle that ABC sightings happen, then you should not discredit this person’s evidence simply because of the low probability of it happening to one person three times.

  • “Inconsistent” had a very specific meaning for us in the fraud team: it was when two things presented as fact could not both be true at the same time. Therefore, at least one of them — if not both — must be untrue or at least inaccurate. If an inconsistency was identified, we would try to resolve it by clarifying our understanding of the facts and then, if necessary, challenging the customer to explain the discrepancy.
Sometimes we got amusing explanations for inconsistencies, and occasionally we accepted them. For example, there was a chap who told us his car had been stolen from the car park of a particular pub and that he had been forced to take the bus home. Inquiries showed that although he was a regular, he had not been seen in the pub that night, his car was not shown on the CCTV footage, and there was no bus home at the time that he had alleged. His explanation for the inconsistency was that his car really had been stolen, but from outside his girlfriend’s house. He had not told us this in the first place because his wife had been listening! Once he had verified this new story by providing contact details for the girlfriend and we had spoken to her — oh, so delicately — we accepted that his car really had been stolen.

In the case of Forteana, inconsistencies may sometimes arise from witness error, or from two witnesses experiencing two different phenomena. However, consider a person who is adamant that they saw the LNM during a trip to the Braemar Highland Games one May about 10 years ago. The games are always in September and until this inconsistency is resolved, his account is open to challenge. If they can demonstrate that they visited that area in both May and September of the same year, then the explanation that they had simply misremembered the details may be acceptable. However, if they described it to one person as “serpent like” and to another as “fat and broad, like a turtle,” this inconsistency would be harder to explain away.

  • “Impossible” was the holy grail for us as fraud investigators. “Impossible” means that something cannot possibly have happened at all, even allowing for a reasonable degree of reporting error.
I once saw a claim where a customer, who owned upstairs commercial premises, reported that burglars had climbed the drainpipe and entered the property via an open window, and escaped the same way with some expensive speaker cabinets. In the first instance, this was sufficiently implausible and improbable to attract our attention. It is not a common method of entry or exit, and heavy speaker cabinets are seldom stolen. Taking them out through an upstairs window did not ring true.

What sealed the claim as “impossible” was the size of the window, which was substantially smaller than the speaker cabinets he had described. With no evidence of damage to the downstairs door locks, we had to conclude that either no theft had occurred, or that the thieves must have gained access through an unlocked door. The story about the window and the drainpipe was demonstrably impossible and the claim was rejected.

In the context of Forteana, there is an additional and interesting additional aspect to this. If a report contains one or more details that are impossible, there are two available conclusions:
  • The report is simply untrue. You cannot have seen the ghost of a drowned fisherman weeping by the waterside at Rutland Water when you were a child in the 1950s, because the lake only came into being in 1976. Therefore, I call BS.
  • Or, we need to modify our understanding of what is possible or impossible. For example, there are proven cases of asexual reproduction (parthenogenesis) in several species of snake. Early reports of this may have been regarded as “impossible” but further investigation has shown that our scientific understanding was wrong: it is rare, but it can happen. Similar considerations may apply to other Fortean phenomena such as ball lightning, or dowsing: one day we may develop a scientific understanding of how it works.
I regard myself as an honest sceptic: someone who does not simply accept anomalous reports at face value, but who is prepared to consider the evidence for and against before reaching a provisional conclusion. I am not the sort to set out in a hostile manner to “debunk” something.

If you have a similar attitude to mine, then each time you hear something out of the ordinary, ask yourself, “What is it about this report that is implausible, improbable, inconsistent, or impossible?”

Implausible will help you to identify your own preconceptions; It helps to remember that improbable only means “unlikely” and therefore implies “possible”; inconsistent gives a focus to your further inquiries; and impossible either means that something has been conclusively disproven, or that there is a potential new field of knowledge to discover.
 
You have just reminded me of one of my top five crime novels of all time - the awesome Don Winslow's, California Fire and Life.

As I've said quite recently elsewhere, of my top five pieces of crime fiction one is about a golf caddy (Pete Dexter's, Train) and another - Winslow's novel - about an insurance claim investigator; jobs which are maybe not the first thing that might come to mind when imagining a great crime novel.

I reckon it's time for a re-read.
 
"Improbable" is a word oft used by people who haven't studied probability.
As in your example, your chance of winning the lottery is minuscule, but the chance of one person among millions of entrants winning is close to 1. I've also had to explain, repeatedly, that the results of each drawing are independent and so choosing the same numbers every time has exactly the same chance of winning as choosing a random set of numbers each drawing.

"One chance in a million" means it would occur around 7000 times across the 7 thousand million or so humans across the planet and up in space.

Then there's the I word that generally cannot be used literally: Inconceivable!
 
the results of each drawing are independent and so choosing the same numbers every time has exactly the same chance of winning as choosing a random set of numbers each drawing.

My son (PhD in particle physics) and I (trained statistician) have tried hard to explain this to Techy but he won't have it. Techy reckons if a number hasn't come up for a while it will do soon because. Why? Because. Because why? Because it'll be time for it to come up again.
 
I suppose I can see that. Because if you had 50 balls and you picked one at random, you'd have a 1 in 50 chance of picking yours. So you'd expect your special chosen number to come up once on average every 50 picks? But the thing is, the longer it hasn't been picked for, it doesn't actually make it more likely that you would pick it. So I can see where he's coming from. Am I getting the gist of it?
 
I've also had to explain, repeatedly, that the results of each drawing are independent and so choosing the same numbers every time has exactly the same chance of winning as choosing a random set of numbers each drawing.

I agree, but imagine having set numbers for years, coming to this realisation then deciding to give them up... only to see those numbers come up a few weeks later. :eek:

That's why I always opt for lucky dip!
 
I read somewhere (can't remember where sorry) that actually some of the numbers are "luckier" than others. Ie they have been drawn more often than others so if you want to win, you would be better to choose the popular numbers. I am not sure how true that is but it sounds cool.

Then there's the I word that generally cannot be used literally: Inconceivable!
:D
inigo_montoya.jpg
 
"Improbable" is a word oft used by people who haven't studied probability.
As in your example, your chance of winning the lottery is minuscule, but the chance of one person among millions of entrants winning is close to 1. I've also had to explain, repeatedly, that the results of each drawing are independent and so choosing the same numbers every time has exactly the same chance of winning as choosing a random set of numbers each drawing.

"One chance in a million" means it would occur around 7000 times across the 7 thousand million or so humans across the planet and up in space.

Then there's the I word that generally cannot be used literally: Inconceivable!

I like this, but isn't there an important difference between the concepts of physical probability (that is - mathematical/statistical probability) and evidential probablity (the latter - Bayesian probability...or something...I think)?

In Mikefule's post, I believe that what he might be talking about is actually an evidential probability issue - which has maybe been a bit confused by an example based on physical probability.

That said, the above is based on vague recall; you both clearly know more about the relevant subjects than I do - and I'm more than happy to be informed that I'm talking out of my arse.
 
Last edited:
I read somewhere (can't remember where sorry) that actually some of the numbers are "luckier" than others. Ie they have been drawn more often than others

I wondered about this too. Statistically speaking, there have been enough draws now since the start of the National Lottery that we could look at the frequencies of being chosen and see if the spread is actually even. Well, it turns out it's all there on the Lotto website.

The spread is quite even over the 2574 draws since 1994. No 13 has come up the least (259 times) and No 40 has come up the most (338 times). The most common numbers to come up are: 40, 23, 38, 11, 35/31/30.

Back in the day when I lived in the UK, you chose 6 numbers and a bonus ball from a 1-49 range but now I see that numbers 50-59 are included but have much lower frequencies. I presume that they have increased the range? Doesn't that make it all the more unlikely to win? Or are those numbers a new "extra" feature or Powerball or something?

Here's the link:
https://www.lottery.co.uk/lotto/statistics
 
I wondered about this too. Statistically speaking, there have been enough draws now since the start of the National Lottery that we could look at the frequencies of being chosen and see if the spread is actually even. Well, it turns out it's all there on the Lotto website.

The spread is quite even over the 2574 draws since 1994. No 13 has come up the least (259 times) and No 40 has come up the most (338 times). The most common numbers to come up are: 40, 23, 38, 11, 35/31/30.

Back in the day when I lived in the UK, you chose 6 numbers and a bonus ball from a 1-49 range but now I see that numbers 50-59 are included but have much lower frequencies. I presume that they have increased the range? Doesn't that make it all the more unlikely to win? Or are those numbers a new "extra" feature or Powerball or something?

Here's the link:
https://www.lottery.co.uk/lotto/statistics
Slight misremember there Ringo, in the main National Lottery you have always chosen six numbers but they draw six numbers and the bonus ball. As you correctly say they have increased the numbers you can choose from 49 to 59. Which means it is indeed more unlikely to win but when you do so on average the jackpots are bigger. They only allow them to roll over a certain amount of times and if the jackpot is not won the money is shared between the lower tier winners. They have also put a set amount on wins other than the jackpot and when a trickle down occurs.
 
With the Lottery, putting aside the chances of any particular number coming up, one possibly marginally more lucrative strategy is supposedly to choose numbers over 31.

This is because punters often pick family birthdays and anniversaries. So while you still might not win, if by some chance your numbers do come up there is less danger of having to share your jackpot with another winner.
 
I agree, but imagine having set numbers for years, coming to this realisation then deciding to give them up... only to see those numbers come up a few weeks later. :eek:

That's why I always opt for lucky dip!

I've run a couple of syndicates, one with set numbers and one on lucky dips, and the lucky dip option is far easier to give up.
 
Ha, for some reason I didn't think of looking on the Lotto website.

Hang on, 13 is the unluckiest number? Whoah! :D
There again 13 is lucky for some I guess!
I did win £99 on only four numbers years ago, since then I've always stayed lucky... I don't do it:hapdan:
 
Last edited by a moderator:
I don't buy lottery tickets. I figure that the money I haven't lost will fund my pension faster than a lottery win:D. I also don't particularly like giving the gov't more money than I have to.
 
Part of my job used to be designing training material for a small team investigating household and motor insurance fraud. For those of us who are interested in the sort of Forteana that involves reports of anomalous sightings (UFOs, Bigfoot, Loch Ness Monster (LNM), ghosts etc.) there are some similarities to the thought processes.

I thought I’d share some ideas here for those with a broadly similar approach to Forteana to my own: what I think of as “honestly sceptical” — as distinct from being determined to debunk everything as a hoax or misidentification.

The insurance claims investigator and the honestly sceptical Fortean are both faced with reports that may or may not be true, supported by unusual or unsatisfactory evidence, or sometimes by none. In both cases, they normally have to work to “the balance of probabilities” rather than applying scientific levels of proof or the criminal law standard of “beyond reasonable doubt.”


In the early days of developing training material, I relied on two common maxims that most Forteans will recognise, whether or not they agree with them:
  • “Extraordinary claims require extraordinary evidence.” (Carl Sagan)
  • “Absence of evidence is not evidence of absence.” (Attributed variously)
I soon concluded that these were unreliable.
  • An extraordinary claim requires only sufficient evidence to support it. The standard of evidence does not have to be higher just because something is unusual. In fact, the converse is more nearly true: a claim that is in no sense unusual may often be accepted with minimal evidence or none — unless of course there a specific reason to doubt it.
  • The absence of evidence that would normally be expected to be available is a form of evidence.

Putting these into context:
  • An extraordinary insurance claim may be the theft of a £10,000 diamond ring from a person with low income and with a home and lifestyle consistent with that low income. However, a document showing that the ring was recently inherited, accompanied by a jeweller’s valuation, would be sufficient. That is the same standard of proof that would be required if a wealthy person were claiming for a ring of similar value. However, in both cases, a claim for a £150 ring might be accepted with no evidence other than a consistent description of the item.
  • An extraordinary claim to a Fortean might be a sighting of the LNM. However, 10 seconds of focussed video footage showing clear morphological detail, with background to provide context and scale, and with verifiable provenance and unedited file properties, would be all it would take to persuade us that it was a genuine sighting. That is not actually a lot of evidence.

  • If a customer, however wealthy, claimed for a £10,000 Patek Philippe watch, the absence of a receipt may not be important. However, imagine a customer who could not provide a receipt, or details of where he bought the watch, or a bank statement showing the payment, or the manual, or the certificate of authenticity, or service records, or even a photo of himself wearing it. The insurer would not be entitled to insist on all of these things, but they would be able to argue that he could not provide at least some of these things rendered the claim “not yet valid”.
  • In the case of the LNM, if someone texted me to say that he had been sitting by the loch for 20 minutes watching the monster at close range, but he had no photos or video of it, I would not believe him. However, if he told me he had seen a red kite 10 miles outside its usual range, I would happily take his word for it.
So, having set aside these two maxims as unreliable for analysing potentially fraudulent claims, I devised what I called the “Four I Test.” I asked my colleagues to ask themselves four questions about any claim that they were considering.

In this order: what part of the claim, if any is:
  • Implausible
  • Improbable
  • Inconsistent
  • Impossible

I think this is a potentially useful approach to any anomalous report. Let’s unpack those four words:
  • “Implausible” means that something is not convincing, or does not seem likely. This says more about the person who is unconvinced than it does about the subject they find unconvincing.
For example, a colleague once said to me, “I don’t believe anyone would spend that much on a child’s birthday present. It’s not as if it was a special birthday. I don’t believe them.” I pointed out that this may be how things were in her family, but someone from a different culture, part of the country, or income bracket, might be equally surprised that her own family spent so little on birthday presents.

In the case of Forteana, we are perhaps all guilty of finding things plausible or implausible depending on our own life experiences and preferences. For example, I often go out into the countryside without my smartphone, so perhaps I find it easier to accept that someone who claims to have seen an ABC did not have their phone with them at the time.

  • “Improbable” means that something can happen, but it is very unlikely. The chances of winning the National Lottery are 45,057,074 to 1 if you buy one ticket. It is incredibly improbable that you will win. However, every week or so, someone does.
In the case of an insurance claim, it may seem improbable that a burglary happened at 7:00 in the morning (most are mid afternoon) in the 10 minutes that the householder was down the garden, but it is not impossible.

In the case of Forteana, it may seem improbable that the same person saw an ABC on their first visit to Dartmoor, another on their first visit to the Norfolk Broads, and another on a day trip to the Peak District. However, if you accept in principle that ABC sightings happen, then you should not discredit this person’s evidence simply because of the low probability of it happening to one person three times.

  • “Inconsistent” had a very specific meaning for us in the fraud team: it was when two things presented as fact could not both be true at the same time. Therefore, at least one of them — if not both — must be untrue or at least inaccurate. If an inconsistency was identified, we would try to resolve it by clarifying our understanding of the facts and then, if necessary, challenging the customer to explain the discrepancy.
Sometimes we got amusing explanations for inconsistencies, and occasionally we accepted them. For example, there was a chap who told us his car had been stolen from the car park of a particular pub and that he had been forced to take the bus home. Inquiries showed that although he was a regular, he had not been seen in the pub that night, his car was not shown on the CCTV footage, and there was no bus home at the time that he had alleged. His explanation for the inconsistency was that his car really had been stolen, but from outside his girlfriend’s house. He had not told us this in the first place because his wife had been listening! Once he had verified this new story by providing contact details for the girlfriend and we had spoken to her — oh, so delicately — we accepted that his car really had been stolen.

In the case of Forteana, inconsistencies may sometimes arise from witness error, or from two witnesses experiencing two different phenomena. However, consider a person who is adamant that they saw the LNM during a trip to the Braemar Highland Games one May about 10 years ago. The games are always in September and until this inconsistency is resolved, his account is open to challenge. If they can demonstrate that they visited that area in both May and September of the same year, then the explanation that they had simply misremembered the details may be acceptable. However, if they described it to one person as “serpent like” and to another as “fat and broad, like a turtle,” this inconsistency would be harder to explain away.

  • “Impossible” was the holy grail for us as fraud investigators. “Impossible” means that something cannot possibly have happened at all, even allowing for a reasonable degree of reporting error.
I once saw a claim where a customer, who owned upstairs commercial premises, reported that burglars had climbed the drainpipe and entered the property via an open window, and escaped the same way with some expensive speaker cabinets. In the first instance, this was sufficiently implausible and improbable to attract our attention. It is not a common method of entry or exit, and heavy speaker cabinets are seldom stolen. Taking them out through an upstairs window did not ring true.

What sealed the claim as “impossible” was the size of the window, which was substantially smaller than the speaker cabinets he had described. With no evidence of damage to the downstairs door locks, we had to conclude that either no theft had occurred, or that the thieves must have gained access through an unlocked door. The story about the window and the drainpipe was demonstrably impossible and the claim was rejected.

In the context of Forteana, there is an additional and interesting additional aspect to this. If a report contains one or more details that are impossible, there are two available conclusions:
  • The report is simply untrue. You cannot have seen the ghost of a drowned fisherman weeping by the waterside at Rutland Water when you were a child in the 1950s, because the lake only came into being in 1976. Therefore, I call BS.
  • Or, we need to modify our understanding of what is possible or impossible. For example, there are proven cases of asexual reproduction (parthenogenesis) in several species of snake. Early reports of this may have been regarded as “impossible” but further investigation has shown that our scientific understanding was wrong: it is rare, but it can happen. Similar considerations may apply to other Fortean phenomena such as ball lightning, or dowsing: one day we may develop a scientific understanding of how it works.
I regard myself as an honest sceptic: someone who does not simply accept anomalous reports at face value, but who is prepared to consider the evidence for and against before reaching a provisional conclusion. I am not the sort to set out in a hostile manner to “debunk” something.

If you have a similar attitude to mine, then each time you hear something out of the ordinary, ask yourself, “What is it about this report that is implausible, improbable, inconsistent, or impossible?”

Implausible will help you to identify your own preconceptions; It helps to remember that improbable only means “unlikely” and therefore implies “possible”; inconsistent gives a focus to your further inquiries; and impossible either means that something has been conclusively disproven, or that there is a potential new field of knowledge to discover.
An excellent summary and I was particularly pleased that you singled out Carl Sagan's most often-quoted remark about "extraordinary claims requiring extraordinary evidence." For someone with a scientific background to come up with such an absurd, irrational, and entirely subjective slogan, and then for it to be seized upon by the dishonourable society of sceptics as some kind of magic talisman for dismissing reports of interesting phenomena, speaks volumes about the low level of thinking in our culture today.
 
I've said it before, but I disagree that Fortean type experiences cannot in the main be looked at like insurance claims.
 
The first rule of the syndicate is, don't talk about the syndicate! Unless you wish to find snail trails across your pillow in the morning...
Ha... the culprit might not be Snails, might be 'mucus mucosa!'
 
Back
Top