Ronald Fisher

How to use this course

Ronald Fisher created much of the toolkit of modern data analysis: maximum likelihood, sufficiency, Fisher information, the analysis of variance, randomized experiments, significance tests as they are used in practice, and the exact test. In biology he reconciled Mendel’s genetics with Darwin’s gradual selection and founded, with Sewall Wright and J. B. S. Haldane, the mathematical theory of evolution. He was also a lifelong advocate of eugenics, and in his last years a public critic of the evidence that smoking causes lung cancer. This course treats all of these, the last two from the documentary record.

Each technical lesson opens with a paragraph for the general reader, then gives precise statements, derivations and exercises with folded solutions. Part III discusses Fisher’s eugenics advocacy and his position on smoking, with the primary and secondary sources listed at the end.

Timeline

Harrow and Cambridge, 1890–1912

Ronald Aylmer Fisher was born on 17 February 1890 in East Finchley, north London, the second of twins. His father, George Fisher, was a partner in the fine-art auctioneers Robinson and Fisher. His mother, Kate, died of peritonitis when he was fourteen.

Fisher’s eyesight was very poor from childhood, and he was forbidden to read under artificial light. Accounts of his schooling describe tutors teaching him mathematics without pen and paper, which trained him to think geometrically. He went to Harrow in 1904 and won its Neeld Medal for mathematics in 1906.

Cambridge

In 1909 he won a scholarship to Gonville and Caius College, Cambridge, and in 1912 he graduated with a First in mathematics. He stayed on for a further year of study; F. J. M. Stratton and James Jeans, who taught him the theory of errors and physics, are listed as his advisors. The theory of errors led to his first paper (1912), which proposed fitting a frequency curve by maximizing what he would later call the likelihood.

At Cambridge he also read Karl Pearson’s papers on the mathematics of evolution and became absorbed in eugenics, the movement founded by Francis Galton to improve human heredity. In 1911 he became the founding chairman of the Cambridge University Eugenics Society. The two interests, statistics and the heredity of human populations, stayed joined for the rest of his life.

City, schoolroom and Pearson, 1913–1919

After a summer on a farm in Canada, Fisher worked from 1913 to 1915 as a statistician for the Mercantile and General Investment Company in the City of London. When war came he volunteered repeatedly and was rejected each time because of his eyesight. From 1915 to 1919 he taught mathematics and physics at several public schools, including Bradfield College.

Research continued in the evenings. In 1915 Pearson’s journal Biometrika published Fisher’s derivation of the exact sampling distribution of the correlation coefficient, using a geometric argument in \(n\)-dimensional space (Lesson 04). In 1917 Pearson’s laboratory published a long “cooperative study” that criticized his method, in a form Fisher judged unfair. This began a feud that lasted until Pearson’s death in 1936 and later passed to Pearson’s son Egon.

In 1917 Fisher married Ruth Eileen Gratton Guinness. They took up subsistence farming alongside his teaching. They had two sons and seven daughters, one of whom died in infancy. Their daughter Joan married the statistician George Box and wrote her father’s standard biography.

In 1918 the Transactions of the Royal Society of Edinburgh published The correlation between relatives on the supposition of Mendelian inheritance, which showed that Mendelian genes can produce the continuous variation the biometricians measured (Lesson 08). In 1919 he had two offers: a post under Pearson at the Galton Laboratory, and a temporary job analysing old field data at Rothamsted Experimental Station. He chose Rothamsted.

Rothamsted, 1919–1933

Rothamsted, at Harpenden in Hertfordshire, had run fertilizer experiments on wheat since the 1840s. Its director, John Russell, hired Fisher to see whether the accumulated records could be made to yield more. Fisher stayed fourteen years and turned the station into the world centre of statistical method.

Studying the yields of the Broadbalk wheat field, he introduced the analysis of variance in Studies in crop variation I (1921). Seeing how badly the old experiments had been planned, he developed the principles of design: replication, randomization and local control by blocking, and the Latin square and factorial layouts. In the same years he wrote the theoretical papers on estimation (1922, 1925) that founded mathematical statistics.

His textbook Statistical Methods for Research Workers (1925) gave working scientists recipes, tables and worked examples, with little proof. It went through fourteen editions and spread the \(t\)-test, the analysis of variance and significance testing through biology, agriculture and medicine. William Sealy Gosset, who published as “Student” while working for the Guinness brewery, was a frequent correspondent; Fisher supplied the rigorous derivation of Student’s \(t\)-distribution and extended it. Frank Yates joined him in 1931 and became his closest collaborator.

He was elected a Fellow of the Royal Society in 1929. In 1930 he published The Genetical Theory of Natural Selection, based on genetics he pursued at home in his spare time, including breeding experiments with mice and poultry.

Small samples and the correlation coefficient

For the general reader. Before Fisher, statisticians mostly used large-sample approximations. Experiments in agriculture or medicine often have only a handful of observations, where these approximations fail. Fisher found exact sampling distributions for small samples, using geometry in many dimensions, and invented transformations that make awkward statistics behave like normal variables.

The sample as a point

Let \(x=(x_1,\dots,x_n)\) be independent \(N(\mu,\sigma^2)\). Fisher viewed \(x\) as one point in \(\mathbb R^n\). Its projection onto the diagonal direction \(\mathbf 1/\sqrt n\) is \(\sqrt n\,\bar x\), and the orthogonal residual \(x-\bar x\mathbf 1\) lies in an \((n-1)\)-dimensional subspace. Because the standard normal law in \(\mathbb R^n\) is rotation-invariant, orthogonal components are independent.

The “\(n-1\)” is the dimension of the residual space. Fisher called it the number of degrees of freedom, and his correct count of degrees of freedom for the \(\chi^2\) test in contingency tables (1922) was another point of conflict with Pearson.

The correlation coefficient

For a bivariate normal sample of size \(n\) with correlation \(\rho\), the sample correlation \(r\) is the cosine of the angle between the two centred data vectors in \(\mathbb R^n\). In 1915 Fisher used this to find the exact density of \(r\). It is skewed and its spread depends strongly on \(\rho\). In 1921 he found a transformation that cures both defects.

Likelihood, sufficiency, information

For the general reader. Given data and a family of possible explanations, which explanation should we prefer, and how precisely do the data pin it down? Fisher’s answer was the likelihood: how probable the observed data would be under each candidate value. His 1922 paper defined the basic properties a good estimate should have and showed that maximizing the likelihood delivers them. It is the founding document of mathematical statistics.

Let \(X_1,\dots,X_n\) be i.i.d. with density \(f(x;\theta)\), \(\theta\in\Theta\subseteq\mathbb R\). The likelihood is \(L(\theta)=\prod_i f(X_i;\theta)\), viewed as a function of \(\theta\), and the maximum likelihood estimator \(\hat\theta\) maximizes it. Fisher insisted that likelihood is not a probability distribution for \(\theta\), and distinguished it sharply from the “inverse probability” of Bayes and Laplace.

In On the mathematical foundations of theoretical statistics (1922) he named three problems (specification, estimation and distribution) and three criteria for estimators: consistency, efficiency and sufficiency.

The score is \(U(\theta)=\frac{\partial}{\partial\theta}\log L(\theta)\). Under regularity conditions (support independent of \(\theta\), differentiation under the integral), \(\mathbb E_\theta U=0\), and its variance is the Fisher information:

Derivation. Expand the score around the true \(\theta\): \(0=U(\hat\theta)\approx U(\theta)+(\hat\theta-\theta)U'(\theta)\). By the central limit theorem \(U(\theta)/\sqrt n\to N(0,I_1)\), and by the law of large numbers \(-U'(\theta)/n\to I_1\). So \(\sqrt n(\hat\theta-\theta)\approx\frac{U/\sqrt n}{-U'/n}\to N(0,I_1/I_1^2)\). The inequality was proved in 1945–46 by C. R. Rao, Fisher’s student, and by Harald Cramér.

Significance tests and the lady tasting tea

For the general reader. A significance test asks whether the data would be surprising if nothing interesting were going on. Fisher made this question routine. His most famous illustration, from The Design of Experiments (1935), is a lady who claims she can tell whether milk or tea was poured first. The design of the test, not only the arithmetic, decides whether her claim can be judged.

In Fisher’s scheme a null hypothesis \(H_0\) specifies the distribution of a test statistic \(T\). The significance level attained, now called the \(p\)-value, is the probability under \(H_0\) of a result at least as extreme as the one observed. Fisher suggested 0.05 as a convenient threshold in the 1925 book, while treating \(p\) as a graded measure of evidence against \(H_0\) and not as a rule of action.

The exact test

The lady is given eight cups, four of each kind, in random order, and told there are four of each. Under \(H_0\) (no discrimination) her choice of four “milk-first” cups is a uniformly random 4-subset of 8. More generally:

Conditioning on the margins removes the nuisance parameter (the common success probability), because the margins are sufficient for it under \(H_0\). Randomizing the order of the cups is what makes the hypergeometric law the correct reference: it guarantees that under \(H_0\) every allocation is equally likely, whatever else may influence her judgement.

Analysis of variance and experimental design

For the general reader. Fields are not uniform: soil, drainage and shade vary from plot to plot. Fisher showed how to split the total variation in a set of measurements into parts due to treatments, to known nuisances such as blocks of land, and to chance. He also showed how to lay out an experiment so that this split is valid. His key move was to assign treatments by randomization.

Suppose \(k\) treatments, each applied to \(m\) plots, with yields \(y_{ij}=\mu+\tau_i+\varepsilon_{ij}\), \(\varepsilon_{ij}\) i.i.d. \(N(0,\sigma^2)\). With group means \(\bar y_{i\cdot}\) and grand mean \(\bar y\),

Fisher worked with \(z=\tfrac12\log F\); George Snedecor tabulated \(F\) itself in 1934 and named it in Fisher’s honour. The independence follows from the geometry of Lesson 04: the decomposition is into orthogonal subspaces of \(\mathbb R^{km}\).

Principles of design

• Replication provides an estimate of error.
• Randomization makes that estimate valid. If the experimenter allocates treatments by a random device, systematic soil differences become part of the chance variation the test accounts for.
• Local control (blocking) removes known sources of variation. In randomized blocks each block contains every treatment once. In a Latin square (picture at the top), treatments are arranged so that each appears once in every row and every column, removing gradients in two directions.
• Factorial designs vary several factors at once. Fisher argued that this is more efficient than changing one factor at a time and is the only way to detect interactions.

Mendel meets biometry: the 1918 paper

For the general reader. Around 1900 biologists split into two camps. Mendelians studied discrete traits such as pea colour; biometricians, led by Pearson, measured continuous traits such as height and found correlations between relatives. Many thought the two views were incompatible. Fisher showed that many Mendelian genes of small effect produce exactly the correlations the biometricians had measured. In the same paper he introduced the word variance and the idea of splitting it into parts.

Take one locus with alleles \(A,a\) at frequencies \(p,q=1-p\), random mating and Hardy–Weinberg proportions \(p^2,2pq,q^2\). Let the genotypic values be \(AA:\ +a_0\), \(Aa:\ d\), \(aa:\ -a_0\). Fisher regressed the genotypic value on the number of \(A\) alleles. The slope \(\alpha=a_0+d(q-p)\) is the average effect of substituting \(A\) for \(a\).

Sketch. A parent passes exactly one allele to a child, so the child shares half the parent’s additive (“breeding”) value and none of the dominance deviation, which depends on the pair. Full sibs share both alleles identically by descent with probability \(\tfrac14\), one with probability \(\tfrac12\), none with \(\tfrac14\); they share \(\tfrac12\) of the additive variance and \(\tfrac14\) of the dominance variance. When the trait is a sum of many independent small contributions, the central limit theorem makes it approximately normal, as Galton’s and Pearson’s data showed. The paper was the foundation of quantitative genetics; heritability in the narrow sense is \(h^2=V_A/V_P\).

The genetical theory of natural selection

For the general reader. The Genetical Theory of Natural Selection (1930) put Darwin’s theory on a mathematical Mendelian basis. It stated a “fundamental theorem” about how fast fitness improves, explained why most species produce sons and daughters in equal numbers, and offered a mechanism for extravagant ornaments such as the peacock’s tail. In 1937 Fisher also wrote down the equation for how a favourable gene spreads through space. The book’s final chapters apply the theory to human societies, in support of eugenics (Lesson 11).

Fisher’s statement was cryptic, and for decades it was read as claiming that mean fitness always rises. George Price (1972) and later authors clarified that it concerns only the part of the change due to changes in gene frequency, holding the “environment”, including the genetic background, fixed. In simple models it holds exactly (Exercise 9.2).

The sex ratio

Suppose sons are rarer than daughters in a population. Every child has exactly one father and one mother, so a son then has more expected offspring than a daughter. Parents genetically inclined to produce sons gain, and the son-producing tendency spreads until the ratio is even. If sons and daughters cost differently to raise, the argument equalizes parental expenditure on the two sexes. The argument was anticipated in part by Darwin and by Carl Düsing, but Fisher’s formulation became standard and is an early example of reasoning later formalized as evolutionary game theory.

Runaway sexual selection

In a 1915 paper and in the 1930 book Fisher argued that a female preference for a male trait, once common, can be self-reinforcing: females who choose ornamented males have ornamented sons who are themselves preferred. Preference and trait can then escalate together beyond any survival advantage.

The wave of advance (1937)

Let \(u(x,t)\) be the frequency of an advantageous gene in a population spread along a line, with dispersal coefficient \(D\) and selective advantage \(s\). Fisher proposed

and argued that the gene advances as a travelling wave of speed \(2\sqrt{Ds}\). Andrey Kolmogorov, Ivan Petrovsky and Nikolai Piskunov analysed the same equation in the same year, and it is now called the Fisher–KPP equation.

Discriminants, fiducial inference, quarrels

For the general reader. Fisher’s later statistical work produced one of the first methods of machine classification and one of his few failures: “fiducial” probability, an attempt to obtain probability statements about unknown parameters without prior distributions. It also produced a lasting quarrel with Jerzy Neyman and Egon Pearson about what a statistical test is for.

Linear discriminant analysis (1936)

In The use of multiple measurements in taxonomic problems, published in the Annals of Eugenics, Fisher used four flower measurements collected by Edgar Anderson to separate iris species. The iris data set is still a standard example in machine learning.

Proof. \(J\) is scale-invariant. By Cauchy–Schwarz in the inner product \(\langle u,v\rangle=u^\top\Sigma v\), \(\big(w^\top(\mu_1-\mu_2)\big)^2=\langle w,\Sigma^{-1}(\mu_1-\mu_2)\rangle^2\le(w^\top\Sigma w)\,(\mu_1-\mu_2)^\top\Sigma^{-1}(\mu_1-\mu_2)\), with equality iff \(w\) is parallel to \(\Sigma^{-1}(\mu_1-\mu_2)\).

Fiducial probability

In 1930 Fisher argued that a pivotal quantity can transfer probability from data to parameter. If \(\bar x\sim N(\mu,\sigma^2/n)\) with \(\sigma\) known, then \(\bar x-\mu\sim N(0,\sigma^2/n)\) for every \(\mu\). Fisher read this, once \(\bar x\) is observed, as a “fiducial distribution” \(\mu\sim N(\bar x,\sigma^2/n)\). In this case the resulting intervals coincide with Neyman’s confidence intervals, but their interpretation differs. In multiparameter problems, notably the Behrens–Fisher problem of comparing two normal means with unequal variances, the fiducial argument gives answers that are not confidence procedures and can be non-unique. Most statisticians now regard it as unsuccessful, though it has modern descendants.

Neyman and Pearson

Neyman and Egon Pearson framed testing as a choice between hypotheses, controlling the error of the first kind and maximizing power, the complement of the error of the second kind. Fisher rejected this decision-theoretic view as suited to industrial acceptance sampling and not to scientific inference. After Neyman criticized the analysis of Latin squares at the Royal Statistical Society in 1935, the dispute became personal. Fisher’s Statistical Methods and Scientific Inference (1956) restated his position. Modern practice is a hybrid of both schools that neither side would have fully accepted.

London, Cambridge and eugenics, 1933–1957

In 1933 Pearson retired from University College London, and his department was split. Egon Pearson took statistics; Fisher became Galton Professor of Eugenics and head of the Galton Laboratory, and from 1934 edited the Annals of Eugenics. The two departments shared a building and did not get on. In 1935 he published The Design of Experiments. With Frank Yates he published Statistical Tables for Biological, Agricultural and Medical Research (1938). At the Galton Laboratory he worked on human blood groups, and the Fisher–Race notation for the Rhesus system dates from these years. In 1943 he moved to Cambridge as Arthur Balfour Professor of Genetics. His marriage broke down during the war.

Honours accumulated: the Royal Medal (1938), the Guy Medal in gold of the Royal Statistical Society (1946), the Darwin Medal (1948), election to the US National Academy of Sciences (1948), a knighthood (1952), the Copley Medal (1955) and the Darwin–Wallace Medal (1958). His doctoral students included C. R. Rao, D. J. Finney, Walter Bodmer and Mary Lyon. In 1956 he was elected President of Gonville and Caius College.

Eugenics

Fisher’s eugenics was not a youthful episode. The documented record includes the following.

• He was founding chairman of the Cambridge University Eugenics Society in 1911 and reviewed books for the Eugenics Review during the First World War.
• The last chapters of The Genetical Theory of Natural Selection (1930) argue that in modern societies the social promotion of the less fertile, and the lower fertility of the more successful classes, cause a decline in inherited ability, and treat this as a cause of the fall of civilizations.
• He proposed that family allowances be paid in proportion to the father’s earnings, so as to encourage larger families among the better-off.
• He served on committees of the Eugenics Society, including the committee campaigning to legalize voluntary sterilization. He supported voluntary, not compulsory, sterilization.
• From 1934 he grew disillusioned with the Eugenics Society and formally dissociated himself from it in 1941.
• After the Second World War he wrote in support of the German geneticist Otmar von Verschuer, who had headed eugenic research under the Nazi regime, and corresponded with him for many years.
• He was among the scientists who dissented from UNESCO’s 1950–51 statements on race. In comments published by UNESCO in 1952, he maintained that human groups differ in innate capacity for intellectual and emotional development.

Historians disagree on how to characterize him. Daniel Kevles described him as an anti-racist conservative; Richard J. Evans has argued that his views were indicative of racism. What is not in dispute is that the advocacy was sustained, public, and intertwined with his science.

Smoking, Adelaide, reassessment, 1957–1962

In 1950 Richard Doll and Austin Bradford Hill reported a strong association between cigarette smoking and lung cancer in a case–control study, and in 1954 they reported early results of a prospective study of British doctors. In 1957 the Medical Research Council stated that the relationship was causal.

Fisher, newly retired from his chair, attacked these conclusions. His letter “Alleged dangers of cigarette-smoking” appeared in the British Medical Journal on 6 July 1957, followed by two letters in Nature in 1958, and the pieces were collected with additions in the booklet Smoking: The Cancer Controversy (1959). His main arguments were:

• Confounding by constitution. A common genetic cause might predispose people both to smoke and to develop lung cancer. He cited twin data, from von Verschuer and from Eliot Slater, showing that identical twins resemble each other more in smoking habits than non-identical twins.
• Reverse causation. A precancerous inflammation might itself lead people to smoke.
• Inhalation. In the published data, patients who said they inhaled had fewer lung cancers than expected among smokers; he argued that this was inconsistent with a simple causal story.
• Correlation is not causation, and observational studies could not settle the question.

Fisher proposed no study to test his constitutional hypothesis. In 1959 Jerome Cornfield and colleagues replied that any hidden factor would need to be many times more common among heavy smokers than non-smokers to explain the observed relative risk, an argument that founded modern sensitivity analysis. The scientific consensus that smoking causes lung cancer was confirmed by the reports of the Royal College of Physicians (1962) and the US Surgeon General (1964).

Fisher was a heavy pipe smoker. His Royal Society memoirists, Frank Yates and Kenneth Mather (1963), recorded the suggestion that his work for tobacco firms as a consultant explained his position, and rejected it as misjudging the man. Paul Stolley’s 1991 analysis, When genius errs, argued that Fisher misread the evidence, never engaged with the full body of data, and let personal and ideological commitments shape his conclusions.

Adelaide and death

Fisher left Cambridge in 1959 and moved to Adelaide, South Australia, where his former colleague E. A. Cornish led the CSIRO Division of Mathematical Statistics. He continued research as a CSIRO fellow. He died in Adelaide on 29 July 1962, at 72, from complications after surgery for colon cancer. He is buried in St Peter’s Cathedral, Adelaide.

Legacy and reassessment

• Statistics: likelihood, sufficiency, information, ANOVA, randomization and the exact test are core curriculum. The statistician Anders Hald called him a genius who almost single-handedly created the foundations of modern statistical science.
• Biology: with Sewall Wright and J. B. S. Haldane he founded population genetics, the mathematical core of the modern evolutionary synthesis. Richard Dawkins called him the greatest biologist since Darwin.
• Reassessment: in June 2020 Gonville and Caius College announced the removal of a 1989 stained-glass window commemorating him. Rothamsted renamed a building that bore his name, UCL removed his name from its Centre for Computational Biology, and the Committee of Presidents of Statistical Societies renamed its Fisher Lectureship.

Sources and further reading

Biography

1. J. J. O’Connor and E. F. Robertson, Ronald Aylmer Fisher, MacTutor History of Mathematics.
2. Ronald Fisher, Wikipedia (with extensive references on eugenics and the 2020 renamings).
3. C. H. Smith, Fisher, Ronald Aylmer (1890–1962), chronology.
4. J. Fisher Box, R. A. Fisher: The Life of a Scientist, Wiley, 1978.
5. F. Yates and K. Mather, Ronald Aylmer Fisher 1890–1962, Biographical Memoirs of Fellows of the Royal Society 9 (1963), 91–129.
6. D. J. Kevles, In the Name of Eugenics, Knopf, 1985.
7. A. Hald, A History of Mathematical Statistics from 1750 to 1930, Wiley, 1998.

Smoking and lung cancer

1. R. A. Fisher, Alleged dangers of cigarette-smoking, British Medical Journal, 6 July 1957.
2. R. A. Fisher, Lung cancer and cigarettes, Nature 182 (12 July 1958), 108; Cancer and smoking, Nature 182 (30 August 1958), 596. Transcription, York history of statistics pages.
3. R. A. Fisher, Smoking: The Cancer Controversy, Oliver and Boyd, 1959.
4. J. Cornfield, W. Haenszel, E. C. Hammond, A. M. Lilienfeld, M. B. Shimkin, E. L. Wynder, Smoking and lung cancer: recent evidence and a discussion of some questions, J. National Cancer Institute 22 (1959), 173–203.
5. P. D. Stolley, When genius errs: R. A. Fisher and the lung cancer controversy, American Journal of Epidemiology 133 (1991), 416–425.

Statistics and genetics

1. R. A. Fisher, Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population, Biometrika 10 (1915).
2. R. A. Fisher, The correlation between relatives on the supposition of Mendelian inheritance, Trans. Roy. Soc. Edinburgh 52 (1918).
3. R. A. Fisher, On the mathematical foundations of theoretical statistics, Phil. Trans. Roy. Soc. A 222 (1922).
4. R. A. Fisher, Statistical Methods for Research Workers, Oliver and Boyd, 1925.
5. R. A. Fisher, The Genetical Theory of Natural Selection, Clarendon Press, 1930.
6. R. A. Fisher, The Design of Experiments, Oliver and Boyd, 1935.
7. R. A. Fisher, The use of multiple measurements in taxonomic problems, Annals of Eugenics 7 (1936).
8. R. A. Fisher, The wave of advance of advantageous genes, Annals of Eugenics 7 (1937).
9. T. Dasgupta, N. Pillai, D. Rubin et al., Comments on the Neyman–Fisher controversy and its consequences, arXiv:1409.2675.
10. G. R. Price, Fisher’s “fundamental theorem” made clear, Annals of Human Genetics 36 (1972).