MLB "Stat Guru" Phil Birnbaum Explains Why "Advanced Basketball Statistics" Don't Work
I have written several articles detailing the flawed methodologies of "advanced basketball statistics," including Economics is Not a Science, Nor is Basketball Statistical Analysis and Economics is Not a Science, Nor is Basketball Statistical Analysis, Part II. Phil Birnbaum is a "stat guru" who primarily focuses on baseball, a sport whose discrete, one on one encounters between pitchers and batters lends itself much more readily to accurate statistical analysis than a free flowing five on five sport like basketball. Birnbaum has taken a look at "advanced basketball statistics" and he is not impressed by what he found:You know all those player evaluation statistics in basketball, like "Wins Produced," "Player Evaluation Rating," and so forth? I don't think they work. I've been thinking about it, and I don't think I trust any of them enough put much faith in their results.
That's the opposite of how I feel about baseball. For baseball, if the sportswriter consensus is that player A is an excellent offensive player, but it turns out his OPS is a mediocre .700, I'm going to trust OPS. But, for basketball, if the sportswriters say a guy's good, but his "Wins Produced" is just average, I might be inclined to trust the sportswriters.
I don't think the stats work well enough to be useful.
Please click on the above link and read Birnbaum's article in its entirety, because he does an excellent job of explaining exactly how difficult it is to correctly assign individual credit for team success in basketball--and Birnbaum does not even address an issue that I have brought up several times: the raw box score numbers themselves are very subjective (I have mainly focused on assists but the same could be said for blocked shots, steals and, to some degree, even rebounds, depending on how the official scorekeepers define tips, etc.).
Birnbaum cites a study by David Lewin and Dan T. Rosenbaum that shows that minutes played by players in a preceding season is at least as good of a predictor of team performance in the subsequent season as the so-called "advanced basketball statistics" are. Birnbaum notes that minutes played "is probably the closest representation you can get to what the coach thinks of a player's skill," so this is an indication that--contrary to the constant bleating by "stat gurus" like Dave Berri and their media sycophants like Henry Abbott--NBA coaches actually do have some idea about what they are doing.
Birnbaum expresses some hope that plus/minus statistics could be useful if the sample sizes are large enough but, as I previously reported, "stat guru" Ken Pomeroy has studied plus/minus stats and is very skeptical of their usefulness. Birnbaum concludes, "But, just picking up a box score or looking up standard player stats online, and trying [to figure out] from that which players are how much better than others (the approach that 'Wins Produced' and other stats take)...well, I don't think you're ever going to be able to make that work."
Labels: "advanced basketball statistics", Dave Berri, Henry Abbott, Ken Pomeroy, Phil Birnbaum
posted by David Friedman @ 5:24 PM
24 Comments:
(1) I suspect that teams with lots of resources are doing some pretty innovate things with advanced statistics that don't rely on box score numbers, but they don't share those things with the public.
(2) Especially when it comes to football, I've always felt that any player evaluation is incomplete without knowing what the player is being coached to do. For example, a linebacker may appear to be out of position on a play, but he might be in the exact spot he was supposed to be at, and on the flip side, a defender might make a spectacular tackle because he freelanced and put the overall defense at risk. Obviously this goes on in basketball as well, and it goes without saying that the stat community doesn't have this on their radar. I have no idea how to account for it (other than trying to infer what is being coached from what I see), but I know it's there, and I know it helps to explain how rebounds are divvied up, how teams balance rebounding vs. fast breaks, how defenses work, etc.
I suppose when it comes down to it I'm not as hostile to the stat guys as you are. They're just fans, they watch tons of games, and they're passionate about basketball. As always, I enjoy reading your blog, and I look forward to reading your thoughts as we approach the playoffs.
That was a very informative read, David. The segment on field goal percentages was particularly well reasoned and very much in line with what you have been saying for years.
Basketball, at its core, is a game with too many variables to be accurately captured by statistical models. It is almost impossible to quantify all the movements that make up even one possession. Not to say that there can never be a reliable metric to measure a basketball player's performance, but we are far from that point.
What do you make about the Lakers' defensive scheme this year? Forcing opponents to make the mid range jumpers?
Anyways, thanks for your high quality site. I've always enjoyed your articles, and I have to say, your writing style and form is excellent. Clear, well reasoned, and always supported by facts and analytical rigor. Keep up the great work!
Very interesting article and I could not agree more.This has been a major problem in all of the socia sciences for decades - group actions are so complex that measuring them successfully is a very difficult, perhaps even impossible task. This is one major reason why qualitative research has become so popular in the last 2 decades - in the examination of complex phenomena - such as human interactions - statistics are not an ideal tool. It seems that Sometimes observation and understanding are needed.
Even the turnover stat can be highly subjective. I'm a longtime Laker fan, so you can probably understand the pain of having to watch Kwame Brown play for my team for more than zero games. Depending on the city, a pass that Kwame fumbled and turned over to the other team frequently counted as a turnover by the PASSER. The most egregious example was a routine, easy bounce pass from Fisher to Kwame (Fisher on the left wing, Kwame near the pinch post) from a distance of ten feet away that hit Kwame right in his hands. Kwame's defender was playing about six feet behind him. This happened in New Orleans, and the Hornet player picked up the fumbled ball near the top of the key and went in for a layup. Fisher was credited with the turnover.
Kudos to Phil B., Ken P. and you for getting this one right on the money!
Understanding and using statistics properly in the game of baseball is fundamentally dissimilar to how it works - or, doesn't work, in fact - in the game of basketball.
David, I've long contended the problem isn't the formulas, it's the stats themselves. The observations kept in the box score just are very good indicators of what's happening or has happened in the game. But now I hear they have software that can plot each player's position on the court (and the ball) 25 times a second, for a whole game, just by using the game video and the software. While this too can be abused by statisticians, I find this to be a promising step forward in providing meaningful analysis, since the "source" data would be so much more descriptive than mere points, rebounds, assists, block, and things of that nature.
JLK1:
(1) I agree with you that there are likely many teams that are using stats in ways that are not being revealed to the public; that is why it is deceptive when Abbott and others talk about what percentage of NBA teams are using "advanced stats" or the winning percentages of those teams compared to the winning percentages of teams that are not using "advanced stats." I seriously doubt that any successful NBA team is basing its personnel or coaching decisions on that Wages of Wins garbage that Abbott links to incessantly, nor are NBA coaches basing their defensive strategies on Abbott's opinions about who are the best clutch players in the league.
(2) You are quite correct about this. During the period of time that I posted many in depth game recaps here I focused on assignment-oriented information--based on my knowledge of the game and/or interviews with NBA coaches, players, etc.--as opposed to just blindly citing stats or trying to make everything fit into a preconceived storyline (those are two traits that show up repeatedly in far too many game recaps).
I don't think that I am unduly hostile to "stat gurus," either in general or regarding specific individuals. I have consistently offered informed, precise critiques of people who spout nonsense (whether that nonsense is stat-oriented or not) and I have also praised "stat gurus" who take a measured, objective approach to their work (such as Roland Beech and Dan Rosenbaum).
Anonymous:
Yes, Birnbaum's analysis of field goal percentage dovetails with many things that I have written, including my analysis of how playing with Kobe Bryant has helped Pau Gasol to increase his field goal percentage.
I have no problem with "stat gurus" seeking the "Holy Grail" of precise measurement of an individual player's impact on winning; I just disagree vigorously with anyone who suggests that this "Holy Grail" has already been found in the form of WoW, PER or any other metric that has been published to date.
Most if not all good defensive teams try to force their opponents to shoot contested two point jumpers; the shots that you don't want to give up are open threes in the corner (the corner three is shorter than any other three but is obviously still worth three points) and uncontested layups/dunks. What varies, based on personnel/matchups, is how each team tries to accomplish those defensive goals. I am not convinced that the Lakers have made such a huge, dramatic schematic change; it seems like the Lakers are playing harder and the bigs are paying more attention to detail regarding their assignments in screen/roll situations, etc. Bynum also seems to have more mobility, more explosiveness and a better conditioning level, though I still don't think that he is 100% right physically.
Yogi:
Statistics are a tool but they do not provide definitive answers by themselves. I look at many statistics when I evaluate players and teams, but those numbers are just one part of a larger picture. The problem is that we have a lot of economists who have not had much success in their primary field so now they have branched out into sports--and the way for them to get attention/make money is to create a new, definitive formula. It is sadly evident that many of these people are not seeking truth but rather seeking a vehicle for self promotion. That may seem like a harsh statement but how else can one explain the way that so many of these individuals completely ignore the scientific method? They do not propose hypotheses, rigorously test them and then draw conclusions; instead, they make outlandishly definitive statements and then try to shout down anyone who challenges them. A true scientist is very circumspect in his statements--the complete opposite of someone like Berri, who keeps loudly declaring that he could make better personnel decisions than NBA executives do.
The Dude Abides:
Turnovers are absolutely an extremely subjective statistic, not only in how they are recorded but also in terms of how significant individual turnovers are. We know that teams want to have a low overall turnover totals but if you look at the NBA record book you will find that most of the turnover records are held by great players. If Magic Johnson had five turnovers that did not necessarily mean that he had a bad game--or even a sloppy game; he may have also scored 25 points, dished 18 assists and grabbed 10 rebounds: someone who touches the ball dozens of times a game may have a few turnovers but the team is still better off with the ball in his hands than with the ball in the hands of players who cannot create much but will still amass a certain number of turnovers. With Magic running the show, the Lakers did not have an undue number of team turnovers even though Magic often ranked among the individual leaders for most turnovers; if Magic had handled the ball less frequently, his turnover total would have gone down but the team turnover total likely would have increased. For instance, the 1991 Lakers ranked fifth best in the league in fewest turnovers even though Magic led the league with the most individual turnovers that season.
Khandor:
Yes, there is a reason that I have written many articles criticizing "advanced basketball statistics" but I have not similarly blasted "sabrmetrics"; the "sabrmetric" approach is much more fundamentally sound (which is not to say that all of the new baseball stats are optimal but that field is in much better shape than "advanced basketball statistics," a field in which people who understand nothing about the sport are pumped up as experts by media members who also understand nothing about the sport).
Gil Meriken:
It is useful to have more data but the bottom line is that data--whether it comes in the form of visual observation, stats or digitized recording of where players are on the court--is only meaningful when it is properly interpreted by someone who understands the larger issues at hand. Even if we have complete records of where players were we still need to have that data interpreted by someone who understands where the players were supposed to be, what they were supposed to be doing and how the greatness of one player impacted what the other nine players did (by drawing a double team, etc.).
I'll tell you the most misleading team statistic besides total points scored or total points given up. It's Points Off Turnovers. It drives me crazy that they don't limit this stat to points scored in transition directly after a turnover. They even count, as part of this stat, points scored after one team travels or throws the ball out of bounds! Ridiculous.
@David
What do you make of the yearly MIT Sloan Sports Annual Analytics Conference? You got some stat geeks claiming that Advanced Statistics will take over some NBA coaching decisions in the next 20years. These are the same stat geeks that were saying that Derek Fisher shouldn't be the starting point guard of a championship team for the past 2 years, yet both Phil and Kobe swear by Fisher.
Heck, Daryl Morey made a lot of decisions based on advanced statistics yet hasn't been able to field a 50 win team.
Look at what Thiboudeau is doing in Chicago? That team is built traditionally: one dynamic superstar(rose), 2nd all star(Boozer), good perimeter defender--shooter and 3rd star(Deng), great rebounder or interior defender(Noah).
You look at Boston and you see same thing.
Heck Stat geeks swore that Miami(3 top PER players from last year) would dominate the league...
I'm not much of a math or statistics guy, but all of this hype over advanced analytics makes me think of a story I heard that pertains to the manufacturer of extremely high-performing stereo equipment (I have a close relative who is a serious audiophile). There has been quite a lot of controversy in that field over measurements of equipment performance and whether they indicate anything about quality. Anyway, a famous audio engineer once said, "If it measures bad and sounds good, it's good. It is measures good and sounds bad, you measured the wrong things."
To translate this to basketball, if it measures inefficiently but wins, it's a winner. If it measures efficiently but loses, you measured the wrong things.
I can't help but also point out that while we can't be sure that the Lakers and Spurs don't use advanced analytics, those organizations, and their coaches, have had extraordinary success without seeming to use them. Funny how the two most successful teams of the 21st century thus far are the ones who seem to eschew advanced stats.
In my college days, I was a progressive rock fan. My two favorite prog rock bands were from the early 1970s heyday--Pink Floyd and Genesis. My other buds who were into prog rock said Yes was a better band. Jon Anderson had that great voice, Rick Wakeman was amazing on keyboards, and Steve Howe was the best rock guitarist in the world of the ones who were well known. My reply was that all three points may be true, but the teamwork of Genesis and the Floyd was so much better. Yes often sounded like a whole bunch of soloists taking turns, while both Genesis and Pink Floyd wrote and played songs that had a complete flow from beginning to end (also, Genesis and Floyd lyrics told a story, while most Yes songs didn't make any sense without a rolled J).
The Cleveland Cavaliers of 2008-10 and this season's Miami Heat are Yes, while the Lakers and Celtics from 2007-2011 are Genesis/Pink Floyd.
To paraphrase Hayek,
"It seems to me that this failure of the basketball writers to analyze more successfully is closely connected with their propensity to imitate as closely as possible the procedures of the brilliantly successful SABREmetrics movement- an attempt which in our field may lead to outright error. It is an approach which has come to be described as the "scientistic" attitude - an attitude which, as I defined it some thirty years ago, "is decidedly unscientific in the true sense of the word, since it involves a mechanical and uncritical application of habits of thought to fields different from those in which they have been formed."
The Dude Abides:
I think that Jeff Van Gundy has ranted about points off of turnovers (and possibly also fastbreak points); it is not clear how those such stats are tracked or what they really tell us.
Jack F:
I have never been to the conference nor have I followed the proceedings in depth. I have no objection to "stat gurus" meeting with each other and trying to figure out how to improve their methodologies but I don't think that their cause is well served by having Henry Abbott as their unofficial spokesman; Abbott's expository skills are not great and his many biases are flagrantly obvious, so the fact that he so vigorously promotes "advanced basketball statistics" probably turns off some people who otherwise might be interested in the work of people like Rosenbaum and Beech. Dean Oliver should be a good hire for ESPN, assuming that they give him free rein to work the way that he has in the past. While doing a search for something else, I stumbled on a good piece by Oliver that talked about LeBron and Wade dribbling too much, though I identified this problem a while ago and did not need "advanced stats" to figure it out. Still, a thoughtful, measured piece by Oliver beats the garbage that is usually posted at True Hoop by Abbott and his THN cohorts.
MSK:
Unlike many writers who are talking out of an orifice other than their mouth when they speak about how teams use "advanced basketball statistics," I have actually done many face to face interviews about this issue, so I can say that Popovich and most of the coaches from his "tree" (most prominently Mike Brown, who led the Cavs to the 2007 Finals) do not rely on "advanced basketball statistics" when making coaching decisions. Check out this article, which includes quotes from Popovich and Bruce Bowen plus a link to my interview with Coach Brown:
Cavs Bounce Back, Rout Spurs
It is also worth noting that this was one of many articles in which I stressed that the foundations of Cleveland's success were LeBron James, defense and rebounding. The narrative that the mainstream media is force feeding the public states that LeBron singlehandedly carried the Cavs but the world is now seeing that even with two handpicked All-Star teammates the Heat are not as good a team as the teams that LeBron played for in Cleveland. I know that some readers may have been irritated that I kept repeating my mantra about the Cavs' foundations but I am literally just one person swimming against a tide of misinformation, so I thought it to be vitally important to keep hammering that point home. I am confident that history will vindicate the correctness of what I wrote.
The Dude Abides:
That is an interesting analogy. I think that what most people fail to understand is that even though the Heat are more talented on paper than the 2009-2010 Cavs were the Heat are not a better team as a functioning unit. The Cavs had a good rotation of bigs and they had depth at every position. The Heat have three All-Stars--the top two of whom with skill sets that are not complementary and the third of whom is being set up, intentionally or not, to be the scapegoat--but their roster is not well balanced. As I said in a recent article, if we could set it up using a time machine the 2009 or 2010 Cavs would beat the 2011 Heat in a playoff series.
9WO:
Precisely--the methods that worked regarding analyzing baseball players and teams are not the appropriate tools to use when trying to analyze basketball players and teams.
David:
re: "I think that what most people fail to understand is that even though the Heat are more talented on paper than the 2009-2010 Cavs"
How much more talented do you think the current version of the Heat actually is compared to last season's Cavs?
Pos, Miami >/=/< Cleveland
PG, Bibby = Williams
OG, Wade > Parker
SF, James = James
PF, Bosh > Varejao
C, Dampier < O'Neal
======================
PG, Chalmers = Gibson
OG, Jones < West
SF, Miller > Moon
PF, Anthony < Jamison
C, Ilgauskas = Ilgauskas
======================
X1, House = Powe
X2, Howard = Hickson
because I don't really perceive there to be THAT much of a difference in their actual "talent level" overall.
Khandor:
That is a good point, though I was talking about something slightly different. I define "talent" (in this context) as how many elite (top 5 to 10) or All-Star level players a team has, while I define "depth" as how many players a team has that can competently play 15 minutes for a playoff team. The Heat have two top five players plus a perennial All-Star in Bosh, while those Cavs teams had one top five player plus several former/recent All-Stars, none of whom is currently as good individually as Wade or Bosh. However, you are quite correct to point out that many of Cleveland's lesser known/lesser regarded players actually match up quite well with their counterparts on the current Heat team.
Perhaps LeBron should have thought about this before he decided to leave a deep, 60-plus win team to play with two big name players in Miami!
Post a Comment
<< Home