Leave lesson

Statistics · Sampling

1 / 17

Bias & sampling methods

What bias really is — a method that systematically leaves people out, so a bigger sample never fixes it — how to name who is missing, the strengths and weaknesses of random, systematic, stratified and convenience sampling, and how to criticise and rewrite a questionnaire.

Statistics · Sampling

Bias & sampling methods

What bias really is — a method that systematically leaves people out, so a bigger sample never fixes it — how to name who is missing, the strengths and weaknesses of random, systematic, stratified and convenience sampling, and how to criticise and rewrite a questionnaire.

Why it works

A sample is a stand-in. You cannot ask all 46 00046\,000 people in a town, so you ask 200200 and then talk as if what those 200200 said is what the town thinks. That swap is only honest if the 200200 are like the town. Bias is what happens when the way you chose them guarantees they are not.

Be precise about the word. Bias is not "an unlucky sample", and it is not "too few people". It is a property of the method: a method is biased when it systematically over-represents one part of the population, or leaves another part out altogether. Systematic is the key word: it pushes the answer the same way every time.

Why that matters more than sample size. A school has 800800 students, and exactly 240240 of them — 30%30\% — cycle to school. You want to estimate that percentage.

Method A. Stand by the bike racks at 8:30 and ask the first 2020 students you see. Almost every one of them cycles, so your estimate is about 100%100\%. Scale it up and you announce 800800 cyclists when the truth is 240240.

Now try the obvious "fix": ask 200200 students at the bike racks instead of 2020. The estimate is still about 100%100\%. You have not moved towards 30%30\% at all — you have repeated the same wrong measurement ten times. The students who walk, take the bus or get a lift are not standing at the bike racks, and no amount of asking will ever reach them.

Method B. Number all 800800 students on the register and use a random number generator to pick 2020. You might get 44 cyclists (20%20\%) or 88 (40%40\%) — the answer wobbles around 30%30\%. Take 200200 the same way and the wobble shrinks: you would expect roughly 6060 cyclists, close to 30%30\%.

There is the whole idea. Random wobble shrinks as the sample grows; bias does not move at all. So "ask more people" is never a cure for a biased method — it buys you a more confident wrong answer. The only cure is to change who is able to be chosen.

Saying who is left out. In an exam, "the sample is biased" earns nothing — everybody writes that. The mark is for naming, in the context of the question, the group that is missed or over-counted, and why that group would have answered differently. Build the sentence in three parts: who was asked → who was missed → why it matters. "Only students at the bike racks are asked, so students who walk or come by bus are never included — and they are exactly the students who don't cycle, so the estimate comes out far too high."

The named methods.
  • Simple random sampling — number every member of the population from a
complete list, then let random numbers choose. Every member has an equal chance. Strength: unbiased by construction. Weakness: you need a full list of the whole population, and it can be slow.
  • Systematic sampling — put the population in order, pick a random start
among the first kk names, then take every kkth name after it (with kk = population size ÷\div sample size). Strength: fast on a long list and spreads the sample right through it. Weaknesses: if the start is not random it is not a random sample at all; and if the list repeats with the same period as kk, the method keeps picking the same kind of person.
  • Stratified sampling — split the population into the groups that matter
(year groups, male and female, departments), then sample each group at random, in proportion to its size. Strength: every group is guaranteed its fair share, so it is usually the most representative. Weakness: you must know the group sizes in advance, and it is more work.
  • Convenience (opportunity) sampling — ask whoever is easiest: your own class,
the first 2020 people you meet, whoever is in the café. Strength: quick, free, no list needed. Weakness: biased almost by definition, because "whoever happened to be there" is a very particular sort of person.

Criticising a questionnaire. Same discipline: name the flaw, then say what it does to the answers.
  • Leading question — "Don't you agree that the new library is a great
improvement?" tells the respondent which answer is wanted.
  • Overlapping boxes — 0 – 2 and 2 – 4: somebody who answers exactly 22 can
tick either box.
  • Boxes that miss people out — boxes 1 – 5 and 6 – 10, with nothing for 00
and nothing for 1111 or more. Those people simply cannot answer.
  • No time frame — "How often do you go to the cinema?" Per week? Per year?
Ever?
  • Socially acceptable answers — "How much fruit do you eat?", asked to your
face, collects the answer people wish were true.

A good rewrite fixes all of these at once: neutral wording, a stated time frame, and boxes that neither overlap nor leave anybody out (finish with an open box such as "1010 or more").

Improving a method. A genuine improvement changes who can be chosen: sample from a list of the whole population, at a range of times and places, using random or stratified selection. Anything that still excludes the same people is not an improvement, however many extra forms you hand out.