A Computer Model Reproduces Data, It Does Not Provide It
In a January 17, 2003 lecture at the California Institute of Technology (a portion of which was recently reprinted by The Wall Street Journal), Michael Crichton declared:I take the hard view that science involves the creation of testable hypotheses...
I regard consensus science as an extremely pernicious development that ought to be stopped cold in its tracks. Historically, the claim of consensus has been the first refuge of scoundrels; it is a way to avoid debate by claiming that the matter is already settled. Whenever you hear the consensus of scientists agrees on something or other, reach for your wallet, because you're being had.
Let's be clear: The work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world. In science consensus is irrelevant. What is relevant is reproducible results. The greatest scientists in history are great precisely because they broke with the consensus.
There is no such thing as consensus science. If it's consensus, it isn't science. If it's science, it isn't consensus. Period...
I would remind you to notice where the claim of consensus is invoked. Consensus is invoked only in situations where the science is not solid enough. Nobody says the consensus of scientists agrees that E=mc2. Nobody says the consensus is that the sun is 93 million miles away. It would never occur to anyone to speak that way...
To an outsider, the most significant innovation in the global warming controversy is the overt reliance that is being placed on models. Back in the days of nuclear winter, computer models were invoked to add weight to a conclusion: "These results are derived with the help of a computer model." But now large-scale computer models are seen as generating data in themselves. No longer are models judged by how well they reproduce data from the real world--increasingly, models provide the data. As if they were themselves a reality...
This fascination with computer models is something I understand very well. Richard Feynman called it a disease. I fear he is right. Because only if you spend a lot of time looking at a computer screen can you arrive at the complex point where the global warming debate now stands.
Nobody believes a weather prediction twelve hours ahead. Now we're asked to believe a prediction that goes out 100 years into the future? And make financial investments based on that prediction? Has everybody lost their minds?
What does this have to do with basketball? Some people believe that the sport can be scientifically analyzed by crunching numbers a certain way--but, as Crichton notes, there is a big difference between using a computer model to "add weight to a conclusion" (or, in basketball terms, provide some information about the performance of a player or a team) and using a computer model to "provide the data. As if they were themselves a reality." If you build a computer model of box score data that values rebounding or not turning the ball over or shooting a high percentage and then rank players accordingly, that model will provide some interesting results to consider--but those results are a model; they are not reality and it is possible to tinker with the numbers to produce a another model that will produce completely different rankings. In other words, as I have repeatedly said, Economics is Not a Science, Nor is Basketball Statistical Analysis.
Labels: basketball statistical analysis, Michael Crichton
posted by David Friedman @ 6:30 AM
4 Comments:
I'm don't subscribe to any religion, but still - AMEN.
How can you make a model based on flawed data? It's like trying to construct a house, but getting only certain pieces of the house.
Ok, as many of my previous posts have indicated, you and I broadly agree about stats, models and basketball.
But on climate issues, Michael Crichton was basically a crank who should not be taken seriously. In particular, I find this hard to take seriously:
Consensus is invoked only in situations where the science is not solid enough. Nobody says the consensus of scientists agrees that E=mc2. Nobody says the consensus is that the sun is 93 million miles away. It would never occur to anyone to speak that way...
There is no need to invoke "consensus" about the distance of the sun because there are not scores of cranks lacking any expertise or scientific knowledge who rail against that fact and attempt to manufacture a controversy where there is virtually none within the scientific community. Such cranks do exist as to evolution and climate change. Scientists and rational persons logically invoke the notion of a "consensus" of scientists when they confront a group of non-scientist morons lacking expertise or knowledge (say on the Kansas board of education, or the Dover, PA school board) who insist on introducing religion into science classrooms.
Anyhow, that's rather far afield. But as one who shares your aversion to some of the stat-obsessiveness re: the NBA, I personally would not invoke Crichton in making my arguments. Relying on or favorably citing him only undermines one's credibility in this arena.
J:
You are completely missing the broader point that Crichton made: the distance between the Earth and the Sun is a scientifically verifiable fact and can be confirmed in any number of ways with our present technology (of course, before we had the technology to measure such distances there was in fact a "consensus" that the Earth was the center of the universe). Can you state with the same degree of certainty--not based on a model or on what you think is likely, but certainty--that significant climate change is taking place due to man-made causes, at what rate that change is taking place, the implications of that change and therefore what should be done about this?
That is a hypothetical question for you and other readers to ponder; I am not really interested in debating the pros/cons of the climate change issue here because it simply is not relevant to the discussion of the broader, important issue highlighted by Crichton, namely defining clearly what is and what is not science. Getting back to the main issue, Crichton is saying two things:
1) It is scientifically wrong to declare something "certain" if that something is not a testable hypothesis; not only is this scientifically wrong but it could have vast negative implications for society if large scale decisions are based on incorrectly declaring something to be "certain."
2) A predictive model that is based on certain data programmed into the model is not equivalent to factual data and the predictions of that model can be changed by altering the data that is inserted. In a basketball sense, what this means is if a stat guru creates a formula that places too much of an emphasis on rebounding then when he crunches the numbers the best rebounders will be disproportionately valued over other players.
David,
Personally, I enjoy hearing people who take a critical lense concerning using potentially charged terms like "consensus". Especially when using these types of words has a particular personal agenda behind them.
Eric
Post a Comment
<< Home