"The Formula That Killed Wall Street": A Cautionary Tale for Anyone Who Places Too Much Value on Basketball Statistical Analysis
I have written several articles that are critical of the methods used in basketball statistical analysis, including a post titled Economics is Not a Science, Nor is Basketball Statistical Analysis. Some of the most prominent proponents of basketball statistical analysis are economists by trade and they apply the methodologies and techniques used in that field to try to quantify what happens on the basketball court. Let me make it quite clear that I am entirely in favor of trying to better quantify and measure the effectiveness of basketball players and teams; what I object to is the haughty contentions by some people in this field that they have already succeeded in accurately making such measurements. Basketball statistical analysis provides some interesting tools that can assist anyone who is trying to compare players and teams but there are limitations to what the numbers alone can accurately depict.Basketball statistical analysis and new video technology have already made for a good marriage in terms of helping teams to more easily produce accurate scouting reports depicting the tendencies, strengths and weaknesses of players and teams. It is now possible to quickly break down game film (or, to be precise, game DVD) and catalog what happened on every pick and roll play, every out of bounds play, every postup and so forth. Most if not all teams are already applying some form of statistical analysis to the tendencies that emerge from game footage and this obviously represents a quantum leap forward in terms of game planning. Cavaliers assistant coach Hank Egan called this "corporate knowledge" when I interviewed him more than three years ago and he said that technological improvements have helped to increase the sophistication of defensive play in the NBA.
The problem is when some people invent certain formulas in which they add up some numbers, multiply other numbers by certain factors, subtract some other numbers and then produce a final number that supposedly "rates" a player's overall performance. It should be obvious that this "rating" is limited by several factors: the accuracy of the original boxscore data, whether or not the additions, multiplications and subtractions correctly value what a player does and, perhaps most importantly of all, the fact that not everything that a player does on the court is captured numerically. I have yet to see any of these stat gurus say that Player X is rated 33.4 with a margin of error of +/- 2.5 points; the stat gurus don't even mention a margin of error because they could not begin to calculate one: they are not performing scientific measurements like a biologist or an astrophysicist--they are massaging basketball statistics in a way that they find appealing and that they believe to be correct (or that will produce conclusions that fit in with their own preconceptions and will be easy to market to book publishers or in other forms of media).
It is interesting that, until fairly recently, economists believed that they had created a formula that--as Wired author Felix Salmon writes in Recipe for Disaster: The Formula That Killed Wall Street--"allowed hugely complex risks to be modeled with more ease and accuracy than ever before." Salmon explains that David X. Li's "Gaussian copula function" formula "made it possible for traders to sell vast quantities of new securities, expanding financial markets to unimaginable levels. His method was adopted by everybody from bond investors and Wall Street banks to ratings agencies and regulators. And it became so deeply entrenched—and was making people so much money—that warnings about its limitations were largely ignored." Li's formula provided "a brilliant simplification of an intractable problem," enabling financial analysts to plug in some data and derive "one clean, simple, all-sufficient figure that sums up everything." The problem is that "people used the Gaussian copula model to convince themselves they didn't have any risk at all, when in fact they just didn't have any risk 99 percent of the time. The other 1 percent of the time they blew up. Those explosions may have been rare, but they could destroy all previous gains, and then some." The economic mess that we are all struggling through now is one of those "1 percent explosions."
Nicholas Nassim Taleb, who I quoted in my Economics is Not a Science, Nor is Basketball Statistical Analysis post, told Salmons, "People got very excited about the Gaussian copula because of its mathematical elegance, but the thing never worked. Co-association between securities is not measurable using correlation. Anything that relies on correlation is charlatanism."
Salmon concludes, "In the world of finance, too many quants (quantitative analysts) see only the numbers before them and forget about the concrete reality the figures are supposed to represent. They think they can model just a few years' worth of data and come up with probabilities for things that may happen only once every 10,000 years. Then people invest on the basis of those probabilities, without stopping to wonder whether the numbers make any sense at all."
This is very analogous to the current situation with basketball statistical analysis; too many of its practitioners "see only the numbers before them and forget about the concrete reality the figures are supposed to represent." They do not have the wisdom or humility to admit that their simple, pretty formulas represent just one, limited interpretation of raw data.
A Wired article titled Road Map for Financial Recovery: Radical Transparency Now! contains a parable that very aptly describes the flaws in the methods used by some basketball statistical analysts:
As (Christopher) Cox sees it, that massive computational power has primarily been used by financial engineers, who create abstract models of how the market should operate and make bets based on those models. "You know Borges, the writer?" Cox asks. "He wrote those fantastical short stories. He has one called On Exactitude in Science." The parable tells of a kingdom obsessed with creating a perfect map of itself—an essentially useless quest that leads them to draw a map that is the same size as the territory it is supposed to represent. Cox sees the story as a metaphor for the modern financial industry, which is so obsessed with modeling the market that it has lost sight of the data beneath those models.
Basketball statistical analysts do not yet have all of the necessary data to completely "model" the sport, nor do they fully understand how to use the data that they have. Trying to produce such a model is certainly a worthy task--but I just wish that the people who are working toward this goal would stop declaring "Mission Accomplished!" when the reality is that they are in the beginning or intermediate stages of that mission.
Labels: basketball statistical analysis, Felix Salmon, Nassim Nicholas Taleb, Wired
posted by David Friedman @ 12:07 AM
12 Comments:
Adjusted +/- actually does provide the standard errors of its estimates, so can say there's an x% probability that a player's "true impact" is within +/- y points per 100 possessions of his adj. +/- rating. I feel your criticism is directed more toward made-up formulas like PER and ones based on misguided regressions like Wins Produced than the more sophisticated +/- metrics out there, like the WinVal employed by Mark Cuban and the +/- used by Daryl Morey. Those are based on concrete fact- how the team scoring margin changed because of the player, which cannot be denied, and makes automatic adjustments for opposing lineup strength and teammate quality.
Good article.
It's seems a bit providential in a way. I recently was listening to Taleb speaking about "black swans" at a college, I believe. And at the end he opened up a Q&A and Kevin Kelly (Co-founder of Wired magazine) asked him a question.
I just thought it was a bit odd reading a basketball blog and coming across those two in the same article. Especially since the comparison is a very fitting one.
Anonymous:
You are correct that my criticism is more focused on the metrics that purport to assign one absolute "rating" to each player.
I agree that adjusted +/- can be a useful tool, particularly when used to compare different lineup combinations. I'm not sure if it is a great tool to make individual player evaluations, though it does give a "quick and dirty" measurement. When I mention unadjusted +/- in a post (like my recent one about Marbury) I make sure to provide specific examples that either show why the +/- number is accurate in this case or why it did not really capture what a particular player did.
Stephen:
In this post, I linked to a previous post that mentioned a review of Taleb's book about "black swans." His examples about the fallibility of certain types of econometric analysis also apply to the way that many basketball statistical analysts operate.
David,
Good read.
I would note that there's a major difference at play, in my opinion. In the financial sector:
"people used the Gaussian copula model to convince themselves they didn't have any risk at all, when in fact they just didn't have any risk 99 percent of the time."
In basketball, even the most statistic-minded GM, Daryl Morey, spends a huge chunk of his efforts tracking player tendencies and efficiencies. Like, he can tell you (and Shane Battier) at what spots on the floor Kobe is most efficient and least efficient, which direction to force him, how deep to get a hand in his face. This is all old-school scouting slipped into a spreadsheet.
No NBA GM relies on any single Holy Grail metric. A few analysts do, and a few of these few have a platform. But this isn't like the financial industry, where the entire system adopted a formula and stripped it of its restraining qualities to profit more. This is basketball teams looking to make every effort to understand the game. We're nowhere near a statistical apocalypse in basketball.
Individual statistical evaluations are the trickiest and most mistreated of all, because of the basic problem of "ownership" of the statistic.
Think of it this way: although player X scored the basket, should the full credit for the two points that go on the scoreboard? If not, what percentage is due to his own efforts? What percentage is due to his teammate Player Y? What about his other teammate Player Z? And isn't this proportion different every time? You can go all the way down the line with this thinking for every individual statistic, including rebounds, assists, steals, blocks, et alia.
For this reason, aggregate team statistics are much more credible, as you don't need to assign individual players "credit". But even team statistical analysis has its flaws.
Ziller:
I think that your explanation of how Morey operates is correct. I discussed that subject in great length in my article for PBN about Michael Lewis' recent NYT article.
As I indicated in the post, what you call "old-school scouting slipped into a spreadsheet" is a perfect marriage of scouting techniques/modern DVD technology/basketball statistical analysis techniques. That approach will obviously help players and teams to perform better. I applaud such efforts and have been chronicling them for several years (in my interview with Egan, my old article about the Art and Science of NBA Defense, etc.).
My objection is directed toward people who claim that they can rank all NBA players precisely based on one number, which I consider to be just as fallacious a claim as what the quants were trying to do with the one number derived by the formula cited in the Wired article.
I agree that the league and its teams are not approaching a "statistical apocalypse" but it disappoints me to see so much attention and positive media coverage directed toward stat metrics that are, as Taleb said of the quant formula cited in the Wired article, "charlatanism." I suspect that the teams know better than to rely on those numbers but will the media voters who select MVPs and All-NBA Team members resist the temptation to use those metrics?
Free Cash Flow:
You are correct. Individual contributions are very difficult to accurately quantify. That is why I insist that at this stage basketball statistical analysis is much more useful for building team scouting reports and compiling data about tendencies than it is for constructing individual player ratings.
Good piece, as usual, and as previous comments have indicated, I certainly agree that statistics are more valuable at a team level, as opposed to an individual level.
FreeCashFlow's comment about "ownership" of a statistic is something I've thought about too, and it also sparked these 2 more thoughts about yet further ways in which our current box score statistics are deficient or fail to capture value-added accurately.
First, take a drive-and-kick assist for a 3-pt shot. That counts equally as relatively low-skill-required pass-back on a 2 v 1 fast break for a lay-up. The assist on the 3-pt bucket is obviously more valuable in the strict points-created sense, but it also is a more difficult play -- that is, a rather mediocre player could probably easily notch an assist on a 2 v 1 fast break, but beating one's defender off the dribble, collapsing the defense, spotting the open perimeter shooter, and accurately passing to the shooter is a much rarer skill.
Second, take what I'll call a "Rondo bailout" situation. The shot clock is winding down, and Rondo has an open look (because the defense has largely been disrespecting and neglecting him), which Rondo passes up and sends the ball instead to Pierce or Allen, who in turn is forced to chuck up a low-percentage outside shot to avoid a 24-second violation. Say that happens 4 times during a game. If one of those shots winds up going in, Rondo gets an assist; if Pierce/Allen miss the other 3 late-in-the-shot-clock heaves, then *they* receive the negative statistical effect of a missed shot, when really Rondo is largely responsible for the miss, as he (or a player of truly all-star or near-all-star caliber) should have taken his open look.
Watching the Celtics and Rondo, it remains absolutely incredible to me to think that Rondo was receiving serious consideration as an all-star. Yes, he is an active defender who does well at playing the lanes and picks up an awful lot of steals, and yes his average of 8+ assists is impressive (good for 6th best in the league currently), and his 5+ rebounds is rather good for a PG. And amazingly he is currently shooting over 50% from the field.
But, if you actually watch him play, a very large proportion of his shots are simple lay-ins (which, granted, he often creates with his speed off the dribble), but he frequently passes up open mid-range looks, and when he does try taking those mid-range shots, he often misses, badly. His stats clearly enjoy some boost from playing alongside three Hall-of-Famers in KG, Pierce, and Allen, and his individual box-score numbers look fairly impressive this year. But in no way is his game that of an elite, all-star caliber player.
To follow-up on my post, the NBA.com "hot spots" feature (http://www.nba.com/hotspots/) bears out my descriptive analysis of Rondo: just look at Rondo's "hot spot" as there's really only 1 such spot, right by the basket, where the vast, vast majority of his shots have been taken. The other 2 red areas involve a total of 30 shots (including a very odd 8-9 from one area behind the 3-pt arc, when he's shooting just 34% overall from 3-pt range).
I'd like to link to his actual hot spot chart, but the flash aspect of the NBA's web site seems to make it difficult to generate a unique URL (tho I know I've seen them in the past).
In any event, on virtually no team other than the Celtics could Rondo really get away from declining to take shots from virtually any spot on the court other than right at the basket, and certainly no true All Star's game is so limited. The impressive individual statistics that Rondo is amassing largely result from the truly high quality of his teammates, who make him look good by draining shots when he gives up the ball and who give opposing defenses so many match-up headaches that Rondo can slip free for some easy lay-ups.
J:
I thought that Rondo played at an All-Star level for a stretch--which good NBA players can do sometimes--but that is not the same thing as actually being an All-Star level player.
As for your larger point, I agree with you that it is very difficult to quantify the value of many different kinds of plays. This is a point that I have made repeatedly regarding Kobe Bryant and I think that sometimes readers think that all I am doing is talking about Bryant when what I am really doing is simply using the game's best player as an example to illustrate larger points about how the NBA game works and how it should be analyzed/understood.
This is one of my biggest problem with stat geeks like John Hollinger. Don't get me wrong, he's a great basketball analyst but it seems to me he relies his whole analyst on his PER rating/stats. He doesn't take into account the player, his weakness or strengths.
It kinda reminds me how people were saying how Amare Stoudemire and Kevin Garnett were both better than Tim Duncan a couple years back.
Or how they say guys like Iverson, Arenas and Baron Davis are superstars yet they never look at their horrible shooting percentage. Or that fact that they don't play defense.
Post a Comment
<< Home