All models are wrong

All models are wrong is a common aphorism and anapodoton in statistics; it is often expanded as "All models are wrong, but some are useful". The aphorism acknowledges that statistical models always fall short of the complexities of reality but can still be useful nonetheless. The aphorism originally referred just to statistical models, but it is now sometimes used for scientific models in general.^[1]

The aphorism is generally attributed to the statistician George Box. The underlying concept, though, predates Box's writings.

Quotations of George Box

 

George Box

The first record of Box saying "all models are wrong" is in a 1976 paper published in the Journal of the American Statistical Association.^[2] The 1976 paper contains the aphorism twice. The two sections of the paper that contain the aphorism state:

2.3  Parsimony
Since all models are wrong the scientist cannot obtain a "correct" one by excessive elaboration. On the contrary following William of Occam he should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity.
2.4  Worrying Selectively
Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about safety from mice when there are tigers abroad.

Box repeated the aphorism in a paper that was published in the proceedings of a 1978 statistics workshop.^[3] The paper contains a section entitled "All models are wrong but some are useful". The section states (p 202-3):

Now it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law PV = nRT relating pressure P, volume V and temperature T of an "ideal" gas via a constant R is not exactly true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of gas molecules. For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No". The only question of interest is "Is the model illuminating and useful?".

Box repeated the aphorism twice more in his 1987 book, Empirical Model-Building and Response Surfaces (which was co-authored with Norman Draper).^[4] The first repetition is on p. 74: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful." The second repetition is on p. 424, which is excerpted below.

... all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind....

A second edition of the book was published in 2007, under the title Response Surfaces, Mixtures, and Ridge Analyses. The second edition also repeats the aphorism twice, in contexts identical with those of the first edition (on p. 63 and p. 414).^[5]

Box repeated the aphorism two more times in his 1997 book, Statistical Control: By Monitoring and Feedback Adjustment (which was co-authored with Alberto Luceño).^[6] The first repetition is on p. 6, which is excerpted below.

It has been said that "all models are wrong but some models are useful." In other words, any model is at best a useful fiction—there never was, or ever will be, an exactly normal distribution or an exact linear relationship. Nevertheless, enormous progress has been made by entertaining such fictions and using them as approximations.

The second repetition is on p. 9: "So since all models are wrong, it is very important to know what to worry about; or, to put it in another way, what models are likely to produce procedures that work in practice (where exact assumptions are never true)".

A second edition of the book was published in 2009, under the title Statistical Control By Monitoring and Adjustment (co-authored with Alberto Luceño and María del Carmen Paniagua-Quiñones). The second edition also repeats the aphorism two times.^[7] The first repetition is on p. 61, which is excerpted below.

All models are approximations. Assumptions, whether implied or clearly stated, are never exactly true. All models are wrong, but some models are useful. So the question you need to ask is not "Is the model true?" (it never is) but "Is the model good enough for this particular application?"

The second repetition is on p. 63; its context is essentially the same as that of the second repetition in the first edition.

Box's widely cited book Statistics for Experimenters (co-authored with William Hunter) does not include the aphorism in its first edition (published in 1978).^[8] The second edition (published in 2005; co-authored with William Hunter and J. Stuart Hunter) includes the aphorism three times: on p. 208, p. 384, and p. 440.^[9] On p. 440, the relevant sentence is this: "The most that can be expected from any model is that it can supply a useful approximation to reality: All models are wrong; some models are useful".

In addition to stating the aphorism verbatim, Box sometimes stated the essence of the aphorism with different words. One example is from 1978, while Box was President of the American Statistical Association. At the annual meeting of the Association, Box delivered his Presidential Address, wherein he stated this: "Models, of course, are never true, but fortunately it is only necessary that they be useful".^[10]

Discussions

There have been varied discussions about the aphorism. A selection from those discussions is presented below.

In 1983, the statisticians Peter McCullagh and John Nelder published their much-cited book on generalized linear models. The book includes a brief discussion of the aphorism (though without citing Box).^[11] A second edition of the book, published in 1989, contains a very similar discussion of the aphorism.^[12] The discussion from the first edition is as follows.

Modelling in science remains, partly at least, an art. Some principles do exist, however, to guide the modeller. The first is that all models are wrong; some, though, are better than others and we can search for the better ones. At the same time we must recognize that eternal truth is not within our grasp.

In 1995, the statistician David Cox commented as follows.^[13]

... it does not seem helpful just to say that all models are wrong. The very word model implies simplification and idealization. The idea that complex physical, biological or sociological systems can be exactly described by a few formulae is patently absurd. The construction of idealized representations that capture important stable aspects of such systems is, however, a vital part of general scientific analysis and statistical models, especially substantive ones, do not seem essentially different from other kinds of model.

In 1996, an Applied Statistician's Creed was proposed by M.R.Nester.^[14] The Creed includes, in its core part, the aphorism.

In 2002, K. P. Burnham and D. R. Anderson published their much-cited book on statistical model selection. The book states the following.^[15]

A model is a simplification or approximation of reality and hence will not reflect all of reality. ... Box noted that "all models are wrong, but some are useful." While a model can never be "truth," a model might be ranked from very useful, to useful, to somewhat useful to, finally, essentially useless.

The statistician J. Michael Steele has commented on the aphorism as follows.^[16]

... there are wonderful models — like city maps....

If I say that a map is wrong, it means that a building is misnamed, or the direction of a one-way street is mislabeled. I never expected my map to recreate all of physical reality, and I only feel ripped off if my map does not correctly answer the questions that it claims to answer.

My maps of Philadelphia are useful. Moreover, except for a few that are out-of-date, they are not wrong.

So, you say, "Yes, a map can be thought of as a model, but surely it would be more precise to say that a map is a 'visually enhanced database.' Such databases can be correct. These are not the kinds of models that Box had in mind."

I agree. ...

In 2008, the statistician Andrew Gelman responded to that, saying in particular the following.^[17]

I take his general point, which is that a street map could be exactly correct, to the resolution of the map.

... The saying, "all models are wrong," is helpful because it is not completely obvious....

This is a simple point, and I can see how Steele can be irritated by people making a big point about it. But, the trouble is, many people don't realize that all models are wrong.

In 2013, the philosopher of science Peter Truran published an essay related to the aphorism.^[18] The essay notes, in particular, the following.

... seemingly incompatible models may be used to make predictions about the same phenomenon. ... For each model we may believe that its predictive power is an indication of its being at least approximately true. But if both models are successful in making predictions, and yet mutually inconsistent, how can they both be true? Let us consider a simple illustration. Two observers are looking at a physical object. One may report seeing a circular disc, and the other may report seeing a rectangle. Both will be correct, but one will be looking at the object (a cylindrical can) from above and the other will be observing from the side. The two models represent different aspects of the same reality.

Truran's essay further notes that Newton's theory of gravitation has been supplanted by Einstein's theory of relativity and yet Newton's theory remains generally "empirically adequate". Indeed, Newton's theory generally has excellent predictive power. Yet Newton's theory is not an approximation of Einstein's theory. For illustration, consider an apple falling down from a tree. Under Newton's theory, the apple falls because Earth exerts a force on the apple—what is called "the force of gravity". Under Einstein's theory, Earth does not exert any force on the apple.^[19] Hence, Newton's theory might be regarded as being, in some sense, completely wrong but extremely useful. (The usefulness of Newton's theory comes partly from being vastly simpler, both mathematically and computationally, than Einstein's theory.)

In 2014, the statistician David Hand made the following statement.^[20]

In general, when building statistical models, we must not forget that the aim is to understand something about the real world. Or predict, choose an action, make a decision, summarize evidence, and so on, but always about the real world, not an abstract mathematical world: our models are not the reality—a point well made by George Box in his oft-cited remark that "all models are wrong, but some are useful".

In 2016, P. J. Bickel and K. A. Doksum published the second volume of their book on mathematical statistics. The volume includes the quote from Box's Presidential Address, given above. It states that the quote is the best formulation of the "guiding principle of modern statistics".^[21]

Additionally, in 2011, a workshop on model selection was held in The Netherlands. The name of the workshop was "All models are wrong...".^[22]

Historical antecedents

Although the aphorism seems to have originated with George Box, the underlying concept goes back decades, perhaps centuries. Some exemplifications of that are given below.

In 1960, Georg Rasch said the following.

... no models are [true]—not even the Newtonian laws. When you construct a model you leave out all the details which you, with the knowledge at your disposal, consider inessential.... Models should not be true, but it is important that they are applicable, and whether they are applicable for any given purpose must of course be investigated. This also means that a model is never accepted finally, only on trial.

— Rasch, G. (1960), Probabilistic Models for Some Intelligence and Attainment Tests, Copenhagen: Danmarks Paedagogiske Institut, pp. 37–38; republished in 1980 by University of Chicago Press

In 1947, the mathematician John von Neumann said that "truth ... is much too complicated to allow anything but approximations".^[23]

In 1942, the French philosopher-poet Paul Valéry said the following.^[24]

Ce qui est simple est toujours faux. Ce qui ne l'est pas est inutilisable.	What is simple is always wrong. What is not is unusable.^[25]
—Valéry, Paul (1942), Mauvaises pensées et autres, Paris: Éditions Gallimard

In 1939, the founder of statistical process control, Walter Shewhart, said the following.^[26]

... no model can ever be theoretically attainable that will completely and uniquely characterize the indefinitely expansible concept of a state of statistical control. What is perhaps even more important, on the basis of a finite portion of the sequence [X₁, X₂, X₃, ...]—and we can never have more than a finite portion—we can not reasonably hope to construct a model that will represent exactly any specific characteristic of a particular state of control even though such a state actually exists. Here the situation is much like that in physical science where we find a model of a molecule; any model is always an incomplete though useful picture of the conceived physical thing called a molecule.

— Shewhart, W. A. (1939), Statistical Method From the Viewpoint of Quality Control, U.S. Department of Agriculture, p. 19

In 1923, a related idea was articulated by the artist Pablo Picasso.

We all know that art is not truth. Art is a lie that makes us realize truth, at least the truth that is given us to understand. The artist must know the manner whereby to convince others of the truthfulness of his lies.

— Picasso, Pablo (1923), "Picasso speaks", The Arts, 3: 315–326;^[27] reprinted in Barr, Alfred H. Jr. (1939), Picasso: Forty Years of his Art (PDF), Museum of Modern Art, pp. 9–12

See also

• Anscombe's quartet – Four data sets with the same descriptive statistics, yet very different distributions
• Bonini's paradox – As a model of a complex system becomes more complete, it becomes less understandable
• Lie-to-children – Teaching a complex subject via simpler models
• Map–territory relation – Relationship between an object and a representation of that object
• Pragmatism – Philosophical tradition
• Reification (fallacy) – Fallacy of treating an abstraction as if it were a real thing
• Scientific modelling – Scientific activity
• Statistical model – Type of mathematical model
• Statistical model validation
• Verisimilitude – Resemblance to reality

Notes

1. Skogen, M.D.; Ji, R.; Akimova, A.; Daewel, U.; and eleven others (2021), "Disclosing the truth: Are models better than observations?" (PDF), Marine Ecology Progress Series, 680: 7–13, Bibcode:2021MEPS..680....7S, doi:10.3354/meps13574, S2CID 229617529.
2. Box, George E. P. (1976), "Science and statistics" (PDF), Journal of the American Statistical Association, 71 (356): 791–799, doi:10.1080/01621459.1976.10480949.
3. Box, G. E. P. (1979), "Robustness in the strategy of scientific model building", in Launer, R. L.; Wilkinson, G. N. (eds.), Robustness in Statistics, Academic Press, pp. 201–236, doi:10.1016/B978-0-12-438150-6.50018-2, ISBN 978-1-4832-6336-6
4. Box, G. E. P.; Draper, N. R. (1987), Empirical Model-Building and Response Surfaces, John Wiley & Sons.
5. Box, G. E. P.; Draper, N. R. (2007), Response Surfaces, Mixtures, and Ridge Analyses, John Wiley & Sons.
6. Box, G. E. P.; Luceño, A (1997), Statistical Control: By Monitoring and Feedback Adjustment, John Wiley & Sons.
7. Box, G. E. P.; Luceño, A.; Paniagua-Quiñones, M. del Carmen (2009), Statistical Control By Monitoring and Adjustment, John Wiley & Sons.
8. Box, G. E. P.; Hunter, W. G. (1978), Statistics for Experimenters, John Wiley & Sons.
9. Box, G. E. P.; Hunter, J. S.; Hunter, W. G. (2005), Statistics for Experimenters (2nd ed.), John Wiley & Sons.
10. Box, G. E. P. (1979), "Some problems of statistics and everyday life", Journal of the American Statistical Association, 74 (365): 1–4, doi:10.2307/2286713, JSTOR 2286713.
11. McCullagh, P.; Nelder, J. A. (1983), Generalized Linear Models, Chapman & Hall, §1.1.4.
12. McCullagh, P.; Nelder, J. A. (1989), Generalized Linear Models (second ed.), Chapman & Hall, §1.1.4.
13. Cox, D. R. (1995), "Comment on "Model uncertainty, data mining and statistical inference"", Journal of the Royal Statistical Society, Series A, 158: 455–456.
14. Nester, M. R. (1996), "An applied statistician's creed" (PDF), Journal of the Royal Statistical Society, Series C, 45 (4): 401–410, doi:10.2307/2986064, JSTOR 2986064.
15. Burnham, K. P.; Anderson, D. R. (2002), Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.), Springer-Verlag, §1.2.5. [As of February 2022,^[update] combined editions of this book have over 60,000 citations on Google Scholar.]
16. Steele, J. M., "Models: Masterpieces and Lame Excuses".
17. Gelman, A. (12 June 2008), "Some thoughts on the saying, "All models are wrong, but some are useful"".
18. Truran, P. (2013), "Models: Useful but Not True", Practical Applications of the Philosophy of Science, SpringerBriefs in Philosophy, Springer, pp. 61–67, doi:10.1007/978-3-319-00452-5_10, ISBN 978-3-319-00451-8.
19. Under Einstein's theory of relativity, the primary reason the apple falls down is that Earth warps time, so that clocks near the base of the tree run more slowly than clocks high up in the tree; there is also a secondary reason, which is that Earth warps space. The empirical evidence for Einstein's theory is extremely strong—e.g. GPS relies on Einstein's theory, and it would not work if it relied on Newton's theory (Ashby 2002).
20. Hand, D. J. (2014), "Wonderful examples, but let's not close our eyes", Statistical Science, 29: 98–100, arXiv:1405.4986, doi:10.1214/13-STS446.
21. Bickel, P. J.; Doksum, K. A. (2016), Mathematical Statistics, vol. II, Chapman & Hall, p. 2.
22. Wit, E.; van den Heuvel, E.; Romeijn, J.-W. (2012), "'All models are wrong...': an introduction to model uncertainty" (PDF), Statistica Neerlandica, 66 (3): 217–236, doi:10.1111/j.1467-9574.2012.00530.x, S2CID 7793470. [See too the workshop web page: "All models are wrong...".]
23. von Neumann, J. (1947), "The mathematician", in Haywood, R. B. (ed.), Works of the Mind, University of Chicago Press, pp. 180–196; republished in 1995 by Bródy F., Vámos T. (editors), The Neumann Compendium, World Scientific, p. 618–626.
24. The relatedness of Valéry's quotation with the aphorism "all models are wrong" has been noted by various authors, e.g. Vankat (2013, §1.7).
25. Some authors have given different English translations, e.g. Valéry (1970, p. 466), Wolfson & Murphy (1998), and Vankat (2013, §1.7). The translation presented here was given by Google Translate; it has only one word different from the translation of Wolfson & Murphy: "What" instead of "Whatever" (both occurrences).
26. The relatedness of Shewhart's quotation with the aphorism "all models are wrong" is noted by Fricker & Woodall (2016).
27. The quotation was originally given in Spanish (during an interview by Marius de Zayas); the cited publication is in English.

References

• Ashby, N. (2002), "Relativity and the Global Positioning System" (PDF), Physics Today, 55 (5): 41–47, Bibcode:2002PhT....55e..41A, doi:10.1063/1.1485583.
• Fricker, R. D., Jr.; Woodall, W. H. (2016), "Play it again, and again, Sam", Significance, 13 (4): 46, doi:10.1111/j.1740-9713.2016.00944.x{{citation}}: CS1 maint: multiple names: authors list (link).
• Valéry, Paul (1970), Collected Works of Paul Valéry, Volume 14—Analects, translated by Stuart Gilbert, Princeton University Press.
• Vankat, J. L. (2013), Vegetation Dynamics on the Mountains and Plateaus of the American Southwest, Springer.
• Wolfson, M. C.; Murphy, B. B. (April 1998), "New views on inequality trends" (PDF), Monthly Labor Review: 3–23.

Further reading

• Anderson, C. (23 June 2008), "The end of theory", Wired
• Box, G. E. P. (1999), "Statistics as a catalyst to learning by scientific method Part II—A discussion", Journal of Quality Technology, 31: 16–29, doi:10.1080/00224065.1999.11979890
• Enderling, H.; Wolkenhauer, O. (2021), "Are all models wrong?", Computational and Systems Oncology, 1 (1): e1008, doi:10.1002/cso2.1008, PMC 7880041, PMID 33585835
• Saltelli, A.; Funtowicz, S. (Winter 2014), "When all models are wrong", Issues in Science and Technology, 30

External links

• "All Models are Right, Most are Useless"—Andrew Gelman blog
• All models are wrong—Peter Coles blog
• Are all models wrong?—Theory, Evolution, and Games blog