Sampling flaws: when the evidence cannot speak for the group

Survey arguments all have the same shape: something was true of the group studied, therefore it is true of a larger group. The entire question is whether the studied group can stand in for the larger one.

So on every one of these, the first move is to name two groups precisely — who was actually studied, and who the conclusion is about. The gap between them is where the answer lives.

The ways a sample fails

• Unrepresentative. The sample differs from the population in a way that bears on the question — surveying gym members about exercise habits and concluding about adults.
• Self-selected. Respondents chose to participate, and the choosing is related to the answer. People with a complaint fill in the complaint form.
• Too small, or of unstated size. A survey with no size is weaker evidence than one with a stated large size, though small does not automatically mean unrepresentative.
• Biased instrument. The question wording, or who asked it, shapes the answer. People report differently to an employer than to an anonymous form.
• Non-response. Those who did not answer differ systematically from those who did, which quietly turns a large sample into a self-selected one.

What answers do

Once you know the two groups, the choices sort themselves quickly.

• Weaken: give a reason the sample differs from the population on the thing being measured.
• Strengthen: say the sample resembles the population in the relevant respect, or close off a specific way it might not.
• Necessary assumption: the sample is representative in the way the argument needs — usually phrased narrowly, and often the least dramatic-sounding choice.
• Flaw: name the mismatch, in the abstract language the test prefers — treats a group as representative when there is reason to think it is not.

The trap on strengthen questions

A very common wrong answer says the sample was large. Size is not representativeness — a hundred thousand people who all chose to respond to an online poll is a hundred thousand self-selected people, and no amount of size repairs the selection.

The complementary trap on weaken questions is an answer attacking the sample on a dimension nobody claimed mattered. If the survey is about commuting times, that the sample skews toward one profession only weakens the argument if profession plausibly relates to commuting.

Do not overcorrect

Not every survey argument is flawed, and a reflex of distrusting all sampling produces wrong answers on Strengthen and Must Be True questions. The test is whether there is a stated or plausible reason the sample differs on the relevant dimension — not whether a sample was used at all.