Justin’s Notes

The Choice of Statistic Is Itself a Degree of Freedom

2026-09-16 · AI-generated insight

McIntyre catalogs the ways social science goes wrong — ideological infection, cherry-picking, fuzzy proxies — and the Lewontin episode that Winegard and Carl recount reads like a case study drawn from his list. Lewontin's famous finding (that most genetic variation lies within populations rather than between them) was not fabricated or sloppy in the ordinary sense. The numbers were right. What was contestable was the choice of statistic: variance at single loci, considered one at a time, rather than correlated structure across many loci. As the face analogy makes vivid, the same underlying data can show near-total overlap or near-total separability depending on whether you look at variables individually or jointly.

This sharpens McIntyre's point about "degrees of freedom" in an important way. He frames cherry-picking mostly as shopping for data — counting immigration costs one way rather than another. But the Lewontin case shows that the analytical frame itself is a degree of freedom, and a more insidious one, because no individual number is wrong. If you know your conclusion in advance, you don't need to touch the data; you only need to pick the lens under which the data yields it. A single-locus analysis and a multivariate one are both "rigorous." They just answer different questions, and which question gets asked can be doing the ideological work.

The synthesis also cuts in the other direction, and honesty requires saying so. McIntyre's warning — "if one knows in advance what one wants to find, one will likely find it" — applies symmetrically. Critics of Lewontin writing in venues invested in reviving race science are no less subject to motivated framing than Lewontin was. The multivariate correction is mathematically real, but what it means — whether statistical clusterability licenses the folk concept of race — is exactly the kind of fuzzy-proxy leap McIntyre flags elsewhere in his list.

The joint lesson: methodological reform of the kind McIntyre calls for cannot just police data and replication. It has to police the quieter choice of which statistic to compute — because that choice can encode the conclusion before a single observation is made.

Woven from these notes

When so many studies fail to be replicated, or draw different conclusions from the same set of facts, it does not instill confidence in the social sciences. Whether this is because of sloppy methodology, ideological infection, or other problems, the result is that even if there are right and wrong answers to many of our questions about human action, most social scientists are not yet in a position to find them. It is not that none of the work in social science is rigorous enough, but when policymakers (and sometimes even other researchers) are not sure which results are reliable, it drives down the status of the entire field. If medicine could break with its barbarous past, isn’t the same path open to the social sciences? For years, many have argued that if they could emulate the “scientific method” of the natural sciences, they too could become more scientific. But this simple advice faces several problems. Among the issues that plague contemporary social scientific research: Too much theory: A number of social scientific studies propose answers that have not been tested against evidence. The classic example here is neoclassical economics, where a number of simplifying assumptions — perfect rationality, perfect information — resulted in beautiful quantitative models that had little to do with actual human behavior. Lack of experimentation/data: Except for social psychology and the newly emerging field of behavioral economics, much of social science still does not rely on experimentation, even where it is possible. For example, it is sometimes offered as justification for putting sex offenders on a public database that doing so reduces the recidivism rate. This must be measured, though, against what the recidivism rate would have been absent the Sex Offender Registry Board (SORB), which is difficult to measure and has produced varying answers. This exacerbates the difficulty in (1), whereby favored theoretical explanations are accepted even when they have not been tested against any experimental evidence. Fuzzy concepts: Some social scientific studies can lead to misleading conclusions because of the use of “proxy” concepts for what one really wishes to measure. A recent example includes measuring “warmth” as a proxy for “trustworthiness,” in which researchers assumed — on the basis of studies which show that we are more likely to trust someone whom we perceive to be “on our side” — that perceptions of scientists as “cold” meant that they would be less trustworthy as well. But the two concepts may not be interchangeable. Ideological infection: This problem is rampant throughout the social sciences, especially on topics that are politically charged. Two ongoing examples are the bastardization of empirical work on the deterrence effect of capital punishment and the effectiveness of gun control on mitigating crime. If one knows in advance what one wants to find, one will likely find it. Cherry picking: The use of statistics allows multiple “degrees of freedom” to scientific researchers, but this is the most likely to be abused. In studies on immigration, for instance, a great deal of the difference between them is a result of alternative ways of counting the “costs” incurred by immigration. This is obviously also related to (4) above. If we know our conclusion, we may shop for the data to support it. Lack of data sharing: As the evolutionary biologist Robert Trivers reports in Psychology Today, there are numerous documented cases of researchers failing to share their data in psychological studies, despite a requirement from APA-sponsored journals to do so. When data were later analyzed, errors were found most commonly in the direction of the researcher’s hypothesis. Lack of replication: Psychology is undergoing a reproducibility crisis. One might validly argue that the initial finding that nearly two-thirds of psychology studies were irreproducible was overblown, but it is nonetheless shocking that most studies are not even attempted to be replicated. This can lead to difficulties, where errors can sneak through. Questionable causation: It is gospel in statistical research that “correlation does not equal causation,” yet some social scientific studies continue to highlight provocative results of questionable value. One recent sociological study, for instance, found that matriculating at a selective college was correlated with parental visitation at art museums, without explicitly suggesting that this was likely an artifact of parental income.
— Lee McIntyre To Fix the Social Sciences, Look to the “Dark Ages” of Medicine

In brief, Lewontin ignored the fact that differences among human populations are correlated. Although population variation at any one genetic locus tends to be small, global population structure becomes clear if one examines correlated differences across loci. For a simple analogy, consider men and women’s faces. If one takes any particular characteristic, say nose size, there is likely to be more variation within a sex than between the sexes (and it would be nearly impossible to classify faces by sex with any accuracy using only nose size). However, if one considers all facial characteristics together (which is, after all, how we actually experience human faces), then sex differences become sufficiently clear that an observer can guess the correct sex more than 95 percent of the time.

— Bo Winegard, Noah Carl Superior: The Return of Race Science—A Review