Introduction

Welcome to the fourth post in this series! In the first one, I introduced my plan for conducting methodological reviews of basketball statistical models. In the second, I reviewed the Wins Produced model. Then, in the last post, I took a look at Win Shares. Today, I will analyze the Wins Above Replacement Player (WARP) model developed by Kevin Pelton.

The Basics of WARP

As Pelton outlines at the beginning of his write-up on WARP, he based much of his work on Dr. Dean Oliver’s. Since I already wrote about Dr. Oliver’s model, I will try not to delve too much into the parts of his writing used by Pelton. Doing so would make this series too repetitive to be ideal as a review.

My opinion is that there are two pivotal frameworks in use by Pelton: Dr. Oliver’s work is the first, while the second is related to his use of the concept of a “replacement player”. Having previously described Oliver’s impact on modern basketball analytics, let’s focus on the effect of the latter framework.

Replacement Level in WARP

According to Fan Graphs, one of the best sites for statistical analysis of baseball: “Replacement level is simply the level of production you could get from a player that would cost you nothing but the league minimum salary to acquire. Minor league free agents, quad-A players, you get the idea. The concept is pretty tidy. These are the players that are freely available and if five of your MLB level players came down with the flu, you could go out and acquire replacement level players without really giving up anything you value other than their union-mandated payday”.

The “replacement-level player” is indeed a tidy concept, but not one I ascribe to for a couple of reasons. First, it is sufficiently vague in its definition to lead to a significant variation on how a modeler may mathematically define replacement level. The result is that different people may have different mental representations of what replacement level is, which increases the noise in the communication regarding the concept. Second, there are other levels besides replacement level to which one can compare players with a model. For instance, the average of a sample has many exciting properties, shown by the sheer amount of times replacement level is defined as a percentage of the mean. A professor of mine once argued that statisticians spend 80% of their time taking unknown values (parameters) and substituting them by the mean of a sample.

The Author

On account of what Kevin Pelton writes about his model, I have a few guesses about the biases he brings to his model. I also looked at his resume and his experiences to further validate my guesses. I do not know the man and have only listened to his participation on several podcasts. As such, I assume that my guesses could well be wrong. However, I hope that they have enough right in them to illuminate the model.

I would wager that most of Pelton’s formal training is in journalism and writing, rather than in maths, statistics, data science, etc. After reading through some of his published works, I arrive at this impression due to the subjects he usually writes about, how he covers them, and the kinds of outlets that feature his coverage.

None of this is meant to be a slight on Pelton. I breach this topic because it usually influences how people create their models. Compared to Dr. Berri and his colleagues and Dr. Oliver, our expectations of Pelton should be different. I would expect his focus to slant more towards creating pieces that reach large audiences and are easy to understand (a skill that I am still trying to develop) rather than rigorous theoretical works. I believe the results are evident in his article, where he describes some of the advantages of using his model: its flexibility, how easy it is to understand, and that it stays constant over time.

The Model

In what follows, I will only focus on what is unique about the WARP model compared to the Win Shares one. Therefore, I will skip to the section where Pelton explains how he estimated how many assisted field goals a player had in the years before the NBA tracked this statistic. He does that by running a regression with multiple variables. In my opinion, his choice of model is problematic.

Assisted Field Goals in WARP

First, he uses linear regression (at least, I assume it is linear since he does not specify) with non-normally distributed variables. Second, the variables he is using significantly correlate with each other. The high correlation means that his regression may have a multicollinearity problem. Furthermore, he is making out of sample predictions with his model. The last point is a problem even in the absence of multicollinearity, and its presence would only increase the likelihood of failure. To predict out of sample is to use data from your sample (not necessarily all of it) to predict outside of it. (Conversely, in-sample predictions mean using parts of the sample to predict other parts of it)

Putting that in context, since Pelton uses data from the years where the NBA accounted for assisted field goals (which make up his sample) to predict assisted field goals in other years (not his sample), he is making out of sample predictions. Unfortunately, out of sample predictions are inherently more difficult, and the adverse effects of multicollinearity heighten in this setting.

Pelton’s use of assisted field goals and a direct relation between usage rate and scoring efficiency are the main differences from the Win Shares model on the offensive side. However, there are many minor ones, like Pelton’s consideration of floor spacing when evaluating players. On defense, he employs statistics that were not tracked when Dr. Oliver wrote Basketball on Paper, in ways similar to what Dr. Oliver would have been expected to use.

Methodological Review

The Good

Kevin Pelton’s work contributions are similar to Justin Kubatko’s on the Win Shares model in that both translated Dr. Oliver’s work into understandable and straightforward models for evaluating player performance. I favor WARP since Pelton added some statistics unavailable to Kubatko and adjustments based on newer basketball understandings. Finally, his translation of offensive and defensive ratings into wins (through Pythagorean estimates of winning percentage) seems more comfortable to understand and agree with than Kubatko’s.

The Biases

I believe everything I wrote about Win Shares’ biases in the last post applies to the WARP model. For that reason, I suggest reading my previous article.

The Questionable

Again, this section would be equal to the one in the Win Shares post, except there is an addition to be made. While I commend Pelton’s work on trying to make his model as complete as possible, he does it through a deterministic lens, which means there’s a higher risk of his model being biased in its results.

The Bottom Line

Like Dr. Oliver and Justin Kubatko, Kevin Pelton made significant contributions to basketball analysis in the public sphere with his model. In fact, from the three models presented in this series so far, the WARP one is probably the one most used in public discussions of basketball.

Up Next

In the next (and penultimate) post of the series, I will methodologically review a series of plus-minus models.