"Stat Gurus" Learn Value of the Eye Test
The idea of using "advanced statistics" to better understand a sport and to more accurately evaluate teams and players has its roots in baseball. Bill James and other pioneers came up with numbers and interpretations that went well beyond the box score. There is certainly a place for that kind of creative thinking. However, the "eye test"--firsthand evaluations made by qualified observers--cannot be replaced by just looking at numbers. "Stat gurus" are often quick to dismiss the "eye test" as subjective and outmoded while at the same time ignoring the inherent limitations of their preferred methodologies.Baseball "stat gurus" are particularly proud of some of the defensive metrics that they have developed. A recent Wall Street Journal article by Andrew Beaton and Michael Salfino termed the quest for accurate defensive numbers "nothing less than the holy grail of baseball statistics." For a while, the "stat gurus" thought that they had solved this problem but now it turns out that some of the much ballyhooed "advanced" defensive statistics are not all that they were cracked up to be, particularly since teams employ defensive shifts so often that it is difficult to precisely determine how much credit a player should receive for a particular defensive play. Beaton and Salfino note that defensive shifts make "it almost impossible to assign proper credit for a great defensive play: Did a player make a play because he has incredible range, or because prescient scouting had him stationed in that area?" Beaton and Salfino conclude, "The result is something that no one would have predicted: Eyeball scouting may be more necessary than ever."
It is obvious that it is important to gather as much useful data as possible and to figure out the best way to interpret that data--but raw numbers (and even "advanced" statistics) cannot take the place of the informed eye. It is like the difference between looking at Cliffs Notes and actually studying the course material; the Cliffs Notes may tell you where to look in a general sense but they do not replace the methodical, step by step learning process. An "advanced" statistic may suggest that a certain baseball player has great defensive range or that a certain basketball player is offensively efficient but it is still essential to watch that baseball player or basketball player in action to place those numbers in context.
Basketball is an inherently more difficult sport to quantify than baseball because basketball involves 10 players moving around at once while baseball is a series of discrete actions. The fact that baseball "stat gurus" are realizing that some of their "advanced" numbers are of limited value should give pause to anyone who blindly relies on "advanced" basketball statistics, particularly regarding defense. If defensive range is hard to quantify in baseball, then it is logical to assume that it is even more difficult to quantify individual defense in basketball; a perimeter player could be horrible defensively and yet his "advanced" numbers may not look bad if he is surrounded by teammates who cover up for his mistakes.
Labels: "advanced basketball statistics", "stat gurus", eye test
posted by David Friedman @ 10:10 AM
