20 Second Timeout

The NBA's Shot-Specific Shooting Percentage Data is Misleading and Meaningless

It should be obvious that "advanced basketball statistics" are only as accurate, meaningful, and relevant as the underlying data. It is easy to document that the underlying data is often incomplete or just flat out wrong. Charley Rosen has provided many examples of this, and several years ago when I tracked Chris Paul's assists it was apparent that his assist totals are inflated. If the box score numbers are skewed then the "advanced basketball statistics" based on those numbers are also skewed. There is no escaping the reality that statistics only provide part of the picture, and thus there is no substitute for watching games with an intelligent eye that can discern skill set strengths and weaknesses to put the numbers in proper context.

There is so much talk about shooting percentages and shooting efficiency, yet so little understanding of the limitations of those numbers, and of how those numbers should be interpreted. You may have seen the proliferation of statistics purporting to track shooting percentages on very specific shots such as step back three pointers. You may have wondered who compiles those shot-specific shooting percentages and how accurate those numbers are. 

It should not be surprising that those numbers are not tracked in any systematic or consistent manner, and consequently they are not accurate:

If you weren't already aware, every shot in the NBA is classified as some version of a jump shot (catch and shoot), pullup jump shot (off the dribble), or increasingly, a step back jump shot in the moment by human scorekeepers. Whenever Steph Curry breaks down his man, steps back, and launches a shot from 28 feet, it’s an error-prone human that determines the type of shot Curry took. 

Although I have never investigated the inaccuracies in the shot-tracking data, I have written about the limitations of "advanced basketball statistics":

Phil Birnbaum has worked extensively with baseball statistics but after thoroughly studying "advanced basketball statistics" he concluded that they are not particularly reliable:

You know all those player evaluation statistics in basketball, like "Wins Produced," "Player Evaluation Rating," and so forth? I don't think they work. I've been thinking about it, and I don't think I trust any of them enough put much faith in their results.

That's the opposite of how I feel about baseball. For baseball, if the sportswriter consensus is that player A is an excellent offensive player, but it turns out his OPS is a mediocre .700, I'm going to trust OPS. But, for basketball, if the sportswriters say a guy's good, but his "Wins Produced" is just average, I might be inclined to trust the sportswriters.

I don't think the stats work well enough to be useful.

Nick Collison is a perfect example of what Birnbaum is talking about. Collison is a plus/minus superstar but does that mean that he is an All-Star or All-NBA caliber player? No, but it could mean any number of other things:

1) Collison very effectively fills a limited role on a team that has two All-NBA players (Kevin Durant and Russell Westbrook) plus a third high quality player (James Harden) who provide scoring and shot creation.

2) Collison is much more effective than other players on his team who play his position, so when he enters the game his team does better than it does with him off of the court.

3) Collison is not better than the other power forwards on his team but he has more of a matchup advantage against the reserve players he competes against than other Thunder power forwards have against the opposing power forwards who they face.

4) Collision's gaudy plus/minus numbers merely reflect a lot of noise due to an insufficiently large sample size of minutes.

I figured out more than a decade ago that I am not going to convince "stat gurus" of the errors of their ways: they are practicing a faith-based religion, not evidence-based science, and thus no amount of evidence will convince them that they are ever wrong. I believe that my readers are more open-minded and intelligent than the "stat gurus," and thus able to understand reality when it is presented in a logical, organized manner.

Labels: "advanced basketball statistics", field goal percentage, NBA

posted by David Friedman @ 1:02 AM

0 comments