Thomas Bayes

How to use this course

Thomas Bayes published almost nothing under his own name. Two tracts appeared anonymously in his lifetime, one on theology and one on the calculus. His fame rests on a manuscript found among his papers after his death, edited by his friend Richard Price and read to the Royal Society in December 1763. That essay solved what was then called the problem of inverse probability: given what has been observed, how probable is each possible cause?

Today his name labels a theorem, a school of statistics, spam filters, and much of machine learning. Much of that is far from what Bayes wrote. This course separates what he did from what was later attached to his name.

Each technical lesson opens with a paragraph for the general reader, then gives precise statements, proofs or sketches, and exercises with folded solutions. Lessons 05 to 08 follow the order of the 1763 essay. Lesson 04 treats his defence of the calculus, and Lessons 09 and 10 treat the two shorter mathematical letters that survive.

The biographical record is thin. Much of what follows depends on the archival work of David Bellhouse (2004) and Andrew Dale (2003). Where a date or attribution is uncertain, the text says so.

Timeline

A dissenting family, 1701–1719

Thomas Bayes was born in 1701 or 1702. No baptism record survives, and the date is reconstructed from the age of 59 on his tombstone. London is the usual birthplace; Andrew Dale noted that the family may then have been in Bovingdon, Hertfordshire, where his father had served a chapel.

His father, Joshua Bayes (1671–1746), was a Presbyterian minister. On 22 June 1694 Joshua was one of six candidates in the first public ordination of dissenting ministers in London since the Act of Uniformity of 1662. He later served the Leather Lane meeting house in Holborn, succeeding Christopher Taylor in 1723, and in 1732 took over the Merchants’ Lecture at Salters’ Hall. Joshua married Anne Carpenter in 1700. Thomas was the eldest of their seven children.

What dissent meant

Protestants who would not conform to the Church of England could worship legally after the Toleration Act of 1689, but they could not graduate from Oxford or Cambridge, which required assent to the Anglican articles. Dissenting families sent their sons to private academies in England or to the Scottish universities. That fact shaped the whole of Bayes’s career. He trained for the ministry in Scotland, served a dissenting congregation, and lived outside the universities.

Schooling

Nothing certain is known of his early education. A 1720 letter to Bayes from John Ward, later professor at Gresham College, suggests a teaching relationship, though Bayes does not appear on Ward’s surviving list of pupils. The story that Abraham de Moivre tutored him is a later suggestion with no documentary support. Bayes certainly read de Moivre’s Doctrine of Chances later, since his essay builds on it.

No reliable portrait exists. The image usually reproduced first appeared in a 1936 book, and historians doubt that it shows him.

Edinburgh, 1719–1722

Bayes was at the University of Edinburgh by February 1719. The surviving records place him in the logic and metaphysics class of Colin Drummond and show that he was admitted to the university library. Bellhouse found that the sponsor for his library admission was James Gregory, professor of mathematics and nephew of the more famous James Gregory of the Gregory series. From this he infers that Bayes also studied mathematics, though no class list proves it.

In 1720 Bayes entered the Divinity Hall, the theological school of the university. Two of his student exegeses survive in the records, on Matthew 7:24–27 (14 January 1721) and on Matthew 11:29–30 (20 January 1722). He left Edinburgh licensed to preach but not ordained; ordination probably followed in London with his father taking part.

Scottish university teaching in the 1710s was moving towards Newtonian natural philosophy. Edinburgh had taught Newton’s ideas since David Gregory held the mathematics chair in the 1680s. A student there could meet fluxions and the Principia at a time when English dissenting academies varied widely in what they offered. The calculus Bayes defended in 1736 is Newton’s, in Newton’s language.

Minister and Fellow, 1722–1752

Back in London, Bayes assisted his father. The first firm London record of him dates from 1728, and by 1732 a list of ministers shows him as his father’s assistant at Leather Lane. In late 1733 or early 1734 he moved to Tunbridge Wells in Kent, a spa town, to succeed John Archer as minister of the Presbyterian congregation at Mount Sion. He remained there for the rest of his life.

Two anonymous tracts

In 1731 he published Divine Benevolence, or an Attempt to Prove That the Principal End of the Divine Providence and Government is the Happiness of His Creatures. It answered the Anglican John Balguy, who had argued that God acts from rectitude; Bayes argued that benevolence is the governing principle. The attribution rests on a reference by Richard Price.

In 1736 came An Introduction to the Doctrine of Fluxions, and a Defence of the Mathematicians Against the Objections of the Author of the Analyst. It answered Bishop George Berkeley’s attack on the calculus (Lesson 04). Ian Hacking judged it perhaps the best reply to Berkeley then available.

The Royal Society

On 25 March 1742 the surgeon John Belchier brought Bayes to a meeting of the Royal Society. His nomination certificate of 8 April was signed by Philip Stanhope, Martin Folkes, James Burrow, Cromwell Mortimer and John Eames. He was elected on 4 November 1742. He had no mathematics in print under his own name. Bellhouse argues that an unpublished paper on trinomial divisors, which survives in Stanhope’s hand, showed his ability to the Fellows who mattered.

Philip Stanhope, second Earl Stanhope, was a mathematical amateur with a wide correspondence, and he became Bayes’s main scientific contact. Stanhope’s papers record a theorem that Bayes told him about at Tunbridge Wells in August 1747, and later letters show Bayes refereeing papers informally for him. A notebook in Bayes’s hand, now held by the Equitable Life Assurance Society, contains notes on probability, series and natural philosophy.

Bayes tried to retire from the ministry in 1749 and finally handed the congregation to William Johnston in 1752. He continued to live in Tunbridge Wells.

Fluxions against Berkeley

For the general reader. In 1734 the philosopher George Berkeley published a witty attack on the calculus. Mathematicians, he said, divide by a small quantity and then set it to zero, which is contradictory; they have no right to mock theologians for believing mysteries. Bayes replied that the calculus does not need infinitely small quantities at all, only the limiting value of a ratio.

The objection

Newton computed the fluxion of \(x^n\) as follows. Let \(x\) become \(x+o\). Then \(x^n\) becomes

so the increments are in the ratio \(o : n o x^{n-1} + \binom n2 o^2x^{n-2}+\cdots\), that is \(1 : n x^{n-1} + \binom n2 o x^{n-2}+\cdots\). Now let \(o\) vanish; the ratio becomes \(1 : nx^{n-1}\). Berkeley pointed out that the argument first assumes \(o\neq0\), to divide by it, and then assumes \(o=0\). He called the vanishing increments the ghosts of departed quantities.

The reply

Bayes based his defence on Newton’s doctrine of prime and ultimate ratios. A fluxion is the velocity with which a quantity grows. The ratio of two fluxions is the ratio that the ratio of the increments approaches as the increments diminish, which it can be brought nearer to than any assigned difference. One never sets \(o=0\) in the ratio. One asks which fixed quantity the ratio can be brought arbitrarily close to.

In this definition \(o\) is never zero, and the statement makes no reference to a final, vanished state. Bayes did not have quantifiers, and his formulation was verbal. The logical structure, a ratio that can be brought within any assigned difference of a fixed value, is the one Augustin-Louis Cauchy and Karl Weierstrass later made exact.

Berkeley’s pamphlet drew several replies, from James Jurin, Benjamin Robins and, most substantially, Colin Maclaurin, whose Treatise of Fluxions (1742) grew out of the controversy. Bayes’s was short and anonymous.

Conditional probability

For the general reader. The first part of Bayes’s essay builds probability from a few definitions. Its key step is a rule for the chance that one thing happened, given that we know another did. In modern form this rule is the statement now called Bayes’s theorem. Bayes himself never wrote it as a single formula.

Bayes’s definitions

Bayes defined the probability of an event as the ratio between the value at which an expectation depending on it ought to be computed and the value of the thing expected on its happening. Probability is a fair betting price. Events are inconsistent if at most one can happen, contrary if exactly one must, and independent if the happening of one neither increases nor decreases the probability of the rest.

Proposition 5 is the decisive move. It asks for the probability of an earlier event given a later one, reversing the order of time in which the events occur. Bayes proved it with a betting argument (Proposition 4): a bet on the first event, made after learning that the second has happened, must be priced so that no one gains from knowing the order of information.

The modern theorem

Proof. By Proposition 3, \(P(H_i\cap E)=P(H_i)P(E\mid H_i)\). The events \(H_j\cap E\) are disjoint with union \(E\), so \(P(E)=\sum_j P(H_j)P(E\mid H_j)\). Divide, using Proposition 5. \(\square\)

For two hypotheses the theorem is cleaner as odds:

posterior odds equal the likelihood ratio (or Bayes factor) times the prior odds. Taking logarithms turns the updating into addition, a fact used by Alan Turing in wartime codebreaking (Lesson 12).

The table and the beta integral

For the general reader. Bayes needed a setting in which a probability is itself unknown but has a known chance of taking each value. He imagined a level square table. One ball is rolled to fix a line at a random place. Further balls are rolled, and each counts as a success if it stops on one side of the line. Seeing only the count of successes, what can we say about where the line is?

The model

Bayes’s postulate: a ball \(W\) thrown on the table is equally likely to stop anywhere, and a line \(os\) is drawn through its resting point parallel to one side. A second ball \(O\) is then thrown \(n\) times, and the event \(M\) is that it stops between the line and the side \(AD\). Write \(\theta\) for the fraction of the table’s width cut off by the line. Then \(\theta\) is uniform on \([0,1]\), and given \(\theta\) the \(n\) throws are independent trials with success probability \(\theta\). Retellings often call this a billiard table; Bayes never used the word.

Bayes stated these as ratios of areas under the curve \(y=x^p(1-x)^q\) drawn over the base of the table. The posterior is the beta distribution with parameters \(p+1\) and \(q+1\). Proposition 10 extends the rule from the table to any unknown event, a step justified in the scholium (Lesson 07).

The normalizing constant

Proof by symmetry. Throw \(n+1=p+q+1\) balls independently and uniformly on the table: one is \(W\), the rest are the \(O\)’s. By symmetry \(W\) is equally likely to be in each of the \(n+1\) positions in left-to-right order, so \(P(\text{exactly }p\text{ of the }O\text{’s lie left of }W)=1/(n+1)\). Conditioning on the position \(x\) of \(W\), the same probability is \(\binom np\int_0^1x^p(1-x)^q\,dx\). Equate and rearrange. \(\square\)

Bayes obtained the same fact by computing the area directly, and recorded it as a corollary: before any ball is thrown, the probability of exactly \(p\) successes in \(n\) trials is \(1/(n+1)\) for every \(p\). The corollary is the hinge of his scholium.

The difficulty in practice was the numerator. The incomplete integral \(\int_b^f x^p(1-x)^q\,dx\) has no closed form, and for large \(n\) the binomial expansion of \((1-x)^q\) produces huge alternating sums. Most of the essay’s second half and Price’s supplement deal with approximating it (Lesson 08).

The scholium and the prior

For the general reader. The table makes the unknown chance uniformly distributed by construction. Real problems do not come with a table. In a short scholium Bayes argued that the same rule should be used whenever we know nothing about an event beforehand. His argument is subtler than the bare assumption that every value is equally likely, and statisticians still debate it.

The argument

Bayes said that an event is unknown when, before any trial, we have no reason to think it should happen any one number of times in \(n\) trials rather than another. Ignorance is a statement about observable counts. By the corollary of Lesson 06, the table model gives exactly this: \(P(S_n=p)=1/(n+1)\) for all \(p\). So, he concluded, the rule of Proposition 9 applies to any unknown event. Stephen Stigler (1982) stressed this reading, which avoids putting a distribution directly on an unobservable parameter.

Proof. Taking \(p=n\), \(E[\theta^n]=P(S_n=n)=\frac1{n+1}=\int_0^1x^n\,dx\) for every \(n\). The uniform distribution has the same moments. A distribution on \([0,1]\) is determined by its moments, because polynomials are uniformly dense in \(C[0,1]\) by the Weierstrass approximation theorem. \(\square\)

Bayes asserted only the forward direction. Some readers think he saw the weakness of assuming a uniform prior directly, which might explain why he left the essay unpublished; Price’s introduction hints that Bayes was unsatisfied with one of his arguments. That remains conjecture.

Exchangeability

A sequence of 0–1 trials is exchangeable if the probability of any finite pattern depends only on its number of 1s. Bruno de Finetti proved in the 1930s that every infinite exchangeable sequence is a mixture of independent trials: there is a distribution \(\mu\) on \([0,1]\) with \(P(X_1=x_1,\dots,X_n=x_n)=\int_0^1\theta^{s}(1-\theta)^{n-s}\,\mu(d\theta)\), where \(s=\sum x_i\). This theorem explains why Bayes’s model, an unknown chance with a prior, is the natural one: it follows from symmetry of the observations alone.

Price’s appendix

Price added worked examples. After \(p\) successes and no failures, the odds that the chance exceeds one half are \(2^{p+1}-1\) to 1. Applied to someone who has seen the sun rise about a million times, the odds that the chance of a sunrise exceeds one half are of the order of \(2^{1\,000\,000}\) to 1. Price also showed that the probability of the chance lying in a narrow interval is much smaller: no finite record gives certainty about the exact proportion.

Approximation for large samples

For the general reader. Bayes’s exact rule needs the area under a curve that cannot be found in closed form. For many trials the curve becomes a tall, narrow bump. Bayes and Price found bounds for its area, and their quantities match what we now call the normal approximation to the posterior.

Bayes’s second rule

Bayes’s rules set \(a=p/n\), \(b=q/n\) and \(m^2=n^3/(pq)\), and estimate the probability that \(\theta\) lies within \(z\) of \(a\) by a series in \(mz\). Price’s 1764 paper proved the rule and supplied sharper bounds. The scale \(m\) is the key. The posterior \(\mathrm{Beta}(p+1,q+1)\) has

Sketch. Write the log density as \(g(x)=p\log x+q\log(1-x)\), maximized at \(\hat x=p/n\). Then \(g''(\hat x)=-p/\hat x^2-q/(1-\hat x)^2=-n^3/(pq)=-m^2\). A second-order Taylor expansion gives \(g(x)\approx g(\hat x)-\tfrac12m^2(x-\hat x)^2\), and the cubic remainder is negligible in the region \(|x-\hat x|=O(1/m)\). This is Laplace’s method, which Laplace developed in the 1770s partly for this very integral.

The general statement, for smooth parametric models and priors positive near the true parameter, is the Bernstein–von Mises theorem: the posterior is asymptotically normal, centred at the maximum-likelihood estimate, with variance the inverse Fisher information divided by \(n\). The prior washes out. This partly answers the worry of Lesson 07.

The divergence of Stirling’s series

For the general reader. James Stirling and Abraham de Moivre found a series for the logarithm of \(z!\) that gives remarkably accurate values. In a letter to John Canton, published with the essay in 1763, Bayes showed that the series cannot be summed: its terms eventually grow without bound for every \(z\). Mathematicians kept using it anyway, and a century later the notion of an asymptotic series explained why that is legitimate.

The series

Bayes wrote the expansion as \(\log z!=\tfrac12\log c+(z+\tfrac12)\log z-\big(z-\tfrac1{12z}+\tfrac1{360z^3}-\tfrac1{1260z^5}+\cdots\big)\), where \(c=2\pi\) is the circumference of a circle of unit radius. In modern notation,

with Bernoulli numbers \(B_2=\tfrac16,\ B_4=-\tfrac1{30},\ B_6=\tfrac1{42},\ B_8=-\tfrac1{30},\ B_{10}=\tfrac5{66}\). The coefficients are \(\tfrac1{12},-\tfrac1{360},\tfrac1{1260},-\tfrac1{1680},\tfrac1{1188},\dots\).

Proof. Euler’s formula \(\zeta(2k)=\dfrac{(-1)^{k+1}B_{2k}(2\pi)^{2k}}{2\,(2k)!}\) with \(1<\zeta(2k)\le\zeta(2)\) gives \(|B_{2k}|\ge 2(2k)!/(2\pi)^{2k}\). So the \(k\)-th term \(t_k\) satisfies

which tends to infinity as \(k\to\infty\) for any fixed \(z\), because the factorial outgrows any geometric sequence. \(\square\)

Bayes did not have Euler’s formula. He derived a recursion for the coefficients and showed directly that the ratio of successive terms eventually exceeds 1 for any \(z\). Euler had noted the divergence of the special case \(z=1\) some years earlier, a fact Bayes did not mention.

Why the series still works

So for fixed \(N\) the error is \(O(z^{-2N-1})\) as \(z\to\infty\). The best accuracy for given \(z\) comes from stopping near the smallest term, roughly at \(k\approx\pi z\). This is the modern definition of an asymptotic series, made precise by Henri Poincaré in 1886. Bayes’s point was the logical one: such a series does not express any quantity as a sum.

Errors that averaging cannot remove

For the general reader. Astronomers in the 1750s were learning to average many observations of the same quantity. Thomas Simpson showed in 1755 that the mean is more likely to be accurate than a single reading. In a letter to Canton, Bayes pointed out a limit. If the instrument itself is imperfect, averaging does not remove that imperfection, however many readings are taken.

Simpson’s argument

Simpson assumed that errors of observation take a few small values symmetrically, and computed the distribution of the error of the mean by counting combinations, much as one counts sums of dice. The mean concentrates near the true value. Bayes questioned the premise. He observed that when an imperfect instrument is used many times, the error of the result tends to stay in proportion to the instrument’s imperfection.

Proof. \(\bar X-\mu=\beta+\bar\varepsilon\) with \(E\bar\varepsilon=0\) and \(\operatorname{Var}\bar\varepsilon=\sigma^2/n\); expand the square. \(\square\)

The random part shrinks like \(1/\sqrt n\); the systematic part does not shrink at all. Every modern measurement budget separates the two, and Bayes’s letter is an early statement of the distinction.

Death, Price and Laplace, 1761–1814

Bayes died at Tunbridge Wells on 7 April 1761. Some sources give 17 April, apparently a clerical error. He was buried in the family vault in Bunhill Fields, the dissenters’ burial ground near Moorgate in London, where his father had been buried in 1746. His will left £200 to be divided between John Hoyle and Richard Price. Bellhouse reads his bequests and friendships as signs that Bayes held Arian views, rejecting the doctrine of the Trinity, as Price did.

Richard Price

Richard Price (1723–1791) was a Welsh dissenting minister at Newington Green, a moral philosopher, and later an authority on life insurance and public finance. Bayes’s relatives gave him the papers. Price worked on the essay for about two years. He wrote an introduction, added footnotes and an appendix of examples, and chose which of Bayes’s solutions to present. On 10 November 1763 he sent it to John Canton, who had it read to the Royal Society on 23 December. It appeared in Philosophical Transactions volume 53 under the title An Essay towards solving a Problem in the Doctrine of Chances. Price’s demonstration of the second rule followed in volume 54.

Price saw the essay as a contribution to natural theology. If regularities in nature can be shown, by calculation, to be unlikely to arise by chance, they point to design. Some modern writers argue that his editorial share is large enough to justify the name Bayes–Price rule.

Laplace

The essay made little immediate impression. In 1774 the young Pierre-Simon Laplace published a memoir on the probability of causes from events, stating the principle that the probabilities of causes are proportional to the probabilities of the observed events under each cause. He knew nothing of Bayes. Laplace treated it as a general method, applied it to astronomy, births and errors of measurement, and developed the asymptotic techniques of Lesson 08. By 1781 he knew Bayes’s essay and accepted its conclusions in a memoir of that year. His Théorie analytique des probabilités (1812) and Essai philosophique sur les probabilités (1814) made inverse probability the standard view for half a century. For most of the nineteenth century the method was associated with Laplace’s name more than with Bayes’s.

How Bayes became Bayesian

Inverse probability lost favour in the nineteenth century. George Boole, in The Laws of Thought (1854), objected that the uniform prior expresses an arbitrary choice rather than ignorance, and others pointed out that a uniform prior on \(\theta\) is not uniform on \(\theta^2\). In the 1920s Ronald Fisher built a theory of inference on likelihood, sampling distributions and significance tests, and rejected inverse probability. Jerzy Neyman and Egon Pearson built the theory of tests and confidence intervals without priors.

The revival came from several directions. Harold Jeffreys’s Theory of Probability (1939) put inverse probability on an axiomatic footing and proposed priors invariant under reparametrization. Frank Ramsey and Bruno de Finetti developed probability as coherent degrees of belief, and de Finetti’s representation theorem (Lesson 07) supplied a rationale for priors. At Bletchley Park during the Second World War, Alan Turing and I. J. Good scored evidence by logarithms of Bayes factors. Leonard Savage’s Foundations of Statistics (1954) gave a decision-theoretic axiom system. Stephen Fienberg traced the adjective “Bayesian” to its spread in the 1950s.

The method remained hard to compute until Markov chain Monte Carlo, devised by physicists in 1953 and brought into statistics around 1990, made posterior integrals tractable in high dimensions. Bayesian methods now run through statistics, machine learning, cosmology, genetics and medicine.

Legacy

• Mathematics: the first rigorous solution of an inverse problem in probability, the beta posterior, and the scholium’s predictive argument for a prior.
• Analysis: an early proof that a useful series diverges, a step towards the theory of asymptotic expansions.
• Measurement: a clear early statement that averaging cannot remove systematic error.
• Name: the International Society for Bayesian Analysis awards the Savage and DeGroot prizes; Bayes’s own name is attached to a theorem he never wrote as a formula and to a school of thought he never knew.

Sources and further reading

Biography

1. D. R. Bellhouse, The Reverend Thomas Bayes, FRS: a biography to celebrate the tercentenary of his birth, Statistical Science 19 (2004) 3–43.
2. J. J. O’Connor and E. F. Robertson, Thomas Bayes, MacTutor History of Mathematics.
3. Thomas Bayes and Joshua Bayes, Wikipedia.
4. I. Hacking, Bayes, Thomas, Dictionary of Scientific Biography.
5. A. I. Dale, Most Honourable Remembrance: The Life and Work of Thomas Bayes, Springer, 2003.
6. S. E. Fienberg, When did Bayesian inference become “Bayesian”?, Bayesian Analysis 1 (2006) 1–40.

Mathematics

1. T. Bayes, An essay towards solving a problem in the doctrine of chances, Phil. Trans. R. Soc. 53 (1763) 370–418. Transcription at York; doi:10.1098/rstl.1763.0053.
2. T. Bayes, A letter to John Canton on asymptotic series, Phil. Trans. R. Soc. 53 (1763) 269–271. Transcription at York.
3. R. Price, A demonstration of the second rule in the essay towards the solution of a problem in the doctrine of chances, Phil. Trans. R. Soc. 54 (1764).
4. [T. Bayes], An Introduction to the Doctrine of Fluxions, and a Defence of the Mathematicians against the Objections of the Author of the Analyst, London, 1736.
5. S. M. Stigler, Thomas Bayes’s Bayesian inference, J. Roy. Statist. Soc. A 145 (1982) 250–258.
6. S. M. Stigler, The History of Statistics: The Measurement of Uncertainty before 1900, Harvard, 1986.
7. A. I. Dale, A History of Inverse Probability from Thomas Bayes to Karl Pearson, 2nd ed., Springer, 1999.
8. C. P. Robert and others, Computing Bayes: Bayesian computation from 1763 to the 21st century, arXiv:2004.06425.
9. H. Jeffreys, Theory of Probability, Oxford, 1939.