The Other Goalie Pull Decision

Uninformative Priors

We have already mentioned the uniform prior on \([0, 1]\), which is equivalently characterized by the \(\text{Beta} \left( 1, 1 \right)\) distribution, as a potential choice of prior. Most likely, the uniform prior is not an accurate representation of the distribution of \(\theta\), as a uniform prior would indicate that we believe that \(\Pr \left( \theta \leq 0.100 \right) = \Pr \left( \theta \geq 0.900 \right) = 0.10\) – in the context of this problem, this would indicate that we believe that our starter allowing over 90% of the shots they face and our starter saving over 90% of the shots they face have equal probability of being true. While this is almost certainly not the case, we may believe that the amount of uncertainty in how our starter is going to perform on a given night is so large that we don’t want to be overconfident in our goalie’s ability for a single game. This is the main benefit of choosing an uninformative prior – if we believe that our goalie’s past performance has little to no effect on how they’re going to perform tonight, then we might want to choose a prior that is not biased toward any one particular outcome, such as the uniform prior that assigns equal likelihood to any possible probability in \([0, 1]\).

In addition, there is another uninformative prior that we might wish to use instead. The \(\text{Beta} \left( 1/2, 1/2 \right)\) prior is shown on the graph below (note that the density goes to infinity at both 0 and 1, but is still a valid probability density):

Unlike the uniform prior, this prior places most of the mass near \(\theta = 0\) and \(\theta = 1\). While it would seem to be more informative than the uniform prior, if we truly knew nothing about the goalie’s save probability or the process of a goalie facing shots on goal, then we might believe that either all shots would be stopped or all shots would go in. More generally, if we are observing a process that ends with either success or failure, but we know nothing else about the process, then we might believe that the process is deterministic before we observe anything – that is, we might believe that the probability of success is either 1 or 0, at least until we see the process produce both success and failure. This prior may be less intuitive, since we know that shutouts are relatively rare and goalies almost certainly won’t let in every shot they face on a given night, but if we want to enter each game with no bias toward the goalie’s past performances, this is another possible choice of uninformative prior.

Data-Driven Priors

Instead of using an uninformative prior, we could use information from previous performances to create a prior that may be more realistic in terms of what distribution a goalie’s save probability might follow. For example, we might want to use the goalie’s most recent game to inform our choice of prior. If the goalie faced \(n\) shots and allowed \(x\) goals, then we might want to choose \(\alpha = 1 + n - x\) and \(\beta = 1 + x\) – that is, we could effectively use the posterior distribution from the previous game (if we started with a uniform prior) as our new prior for the goalie’s next game. (The additional 1 in each parameter is to ensure that neither \(\alpha\) nor \(\beta\) is 0, as would be the case if the goalie posted a shutout in their previous game.) If we wanted to use a larger sample of data, we could also consider the goalie’s previous \(k\) games (where \(k\) is an arbitrary positive integer of our choice), taking the total number of saves and total number of goals allowed as \(\alpha\) and \(\beta\), respectively.

Using this method of selecting a prior would indicate a good deal of confidence in our estimate of \(\theta\), as we would be using a full game’s worth of shots to inform our prior, instead of taking an uninformative prior (recall that larger values of \(\alpha\) and \(\beta\) convey more confidence in our prior beliefs). However, while this would provide a more realistic prior for \(\theta\), it could also be detrimental to our end goal. As an example, if our goalie posted a 50-save shutout in their previous game, the prior for the current game could inspire overconfidence in \(\theta\) being very close to 1, so if the starter then allows 2 goals on 3 shots, our posterior distribution would still be highly confident in the goalie’s ability, even though the evidence for this game quickly appears to be suggesting otherwise. Then again, if our goalie posted a shutout in their previous game, we might be willing to give them more slack in their next game. Using more than one previous game to inform a prior constructed in this way would only serve to provide more confidence in our beliefs, which would amplify both the advantages and disadvantages of choosing a prior in this way.

As an alternative to taking raw save and goal totals to inform our choice of prior, another possible option would be to fit a beta distribution to previously observed values of \(\theta\). Since we are assuming that \(\theta\) is a random variable that is realized as a fixed value for each game a goalie plays, it may be reasonable to record the values of \(\theta\) for several goalie starts and fit a beta distribution to those values. While \(\theta\) is unknown for each game (since under our model assumptions we cannot ever be certain of the goalie’s true save probability), we can obtain a point estimate for \(\theta\) from each game by using Laplace’s rule of succession, which states that if we observe \(n\) shots on goal and \(x\) goals and nothing else, the probability that the next shot would be saved is equal to \((n - x + 1) / (n + 2)\). In fact, this is the mean of the posterior distribution obtained if we observe \(x\) goals on \(n\) shots and assume a uniform prior on \(\theta\). Since we can observe the values of \(n\) and \(x\) for each goalie start, we can consider estimates of \(\theta\) under the rule of succession. In the context of this problem, our estimates for \(\theta\) are close to our usual understanding of save percentage, but we need to modify them in order to account for our uncertainty – we are assuming that the true value of \(\theta\) is still unknown after the game ends, but typically we understand save percentage within a game to be the proportion of shots saved by the goalie, which is always known after the game ends.

Then, given estimates of \(\theta\) (obtained via the rule of succession) from some sample of \(k\) games that we have observed, denoted \(\hat{\theta}_{1}, \dots, \hat{\theta}_{k}\), we can attempt to find the values of \(\hat{\alpha}\) and \(\hat{\beta}\) such that the \(\text{Beta} \left( \hat{\alpha}, \hat{\beta} \right)\) distribution best fits the observed values of \(\hat{\theta}_{1}, \dots, \hat{\theta}_{k}\). Without getting too deep into the statistical theory, there are two main ways to fit a distribution in this way: using the method of moments, or using maximum likelihood estimation.

However, before fitting any distributions, we first need to obtain some data. MoneyPuck.com provides a dataset for download containing information about all unblocked shots beginning in the 2007-2008 season, with 124 features for each shot. For this preliminary exploration, we are only concerned with the number of shots faced and goals allowed by the starting goalies for each team in each game played. We will also restrict our attention to regular season games played from the 2014-2015 season through the 2020-2021 season, and we will only consider shots faced and goals allowed in regulation time (since overtime hockey is played at a far different pace from regulation hockey). From this information, we can compute an estimated save probability for each goalie start. The smoothed density of these estimated save probabilities is shown below:

Given the sample average \(\bar{x}\) and the sample variance \(s^{2}\) of a sample of observations drawn from a \(\text{Beta} \left( \alpha, \beta \right)\) distribution, the method-of-moments estimators \(\hat{\alpha}_{\text{MM}}\) and \(\hat{\beta}_{\text{MM}}\) are computed by

\[\begin{align} \hat{\alpha}_{\text{MM}} &= \bar{x} \left( \frac{\bar{x} \left( 1 - \bar{x} \right)}{s^{2}} - 1 \right) \\ \hat{\beta}_{\text{MM}} &= \left( 1 - \bar{x} \right) \left( \frac{\bar{x} \left( 1 - \bar{x} \right)}{s^{2}} - 1 \right) \end{align}\]

assuming that \(\bar{x} \left( 1 - \bar{x} \right) > s^{2}\). We find that \(\hat{\alpha}_{\text{MM}} \approx 21.322\) and \(\hat{\beta}_{\text{MM}} \approx 2.969\). The method-of-moments density is represented by the dashed line in the plot below, along with the actual density of estimated starter save probabilities:

This fitted distribution is relatively similar to the actual distribution, although the range of \(\theta\) around the mode of the fitted distribution has less mass than in the actual distribution, which is made up for by the fitted density having slightly more mass in the range around \(\theta = 0.80\). For the maximum likelihood estimators of \(\alpha\) and \(\beta\), we cannot obtain closed forms, but we can use any simple statistical software to compute \(\hat{\alpha}_{\text{MLE}} \approx 22.541\) and \(\hat{\beta}_{\text{MLE}} \approx 3.165\), which gives the following distribution:

The shape of this fitted distribution appears to be slightly better than the previous fitted distribution, although the differences do not appear to be significant (as evidenced by the small differences between the values of the two estimators). For these reasons, either method of parameter estimation is likely appropriate for this exercise, although maximum likelihood estimation is typically preferred in general statistical settings.

It is worth noting again that this part of the project is not attempting to find an optimal prior, nor is it attempting to make any suggestions about what we should be using as a prior. With that in mind, either of the priors computed here via the method of moments or via maximum likelihood estimation are reasonable choices of prior. Neither set of estimators is perfect, but they each provide a prior that, in general, provides more confidence in our starter than an uninformative prior, but not as much confidence in our starter as simply using the raw shot and goal totals from their previous game(s). The most important thing to note is that it appears that estimated starter save probabilities do closely follow a beta distribution, and that the method of moments and maximum likelihood estimation both seem to be viable methods of creating a prior.

While we could simply choose the previous \(k\) league-wide starter performances to create a prior in this way, which would give us a fairly large sample size of estimated starter save probabilities, it may be the case that we want to tailor our prior more closely to our team’s goalie(s). In this regard, we could also restrict our estimates \(\hat{\theta}_{k}\) to either the team level or the goalie level. As an extreme example, we would likely expect the distribution of \(\theta\) to be much different for a recent Vezina trophy winner compared to a rookie or aging veteran, so we may want to include only performances from our Vezina-winning goalie in fitting a prior for their next game. Conversely, we also might want to temper our expectations a bit – we know that even Vezina winners have bad games from time to time, so we may want our prior to reflect that possibility as being more likely than just the Vezina winner’s recent games might suggest, in which case we might want to include all goalie starts from across the league.

With all of this in mind, the upshot here is that there are several possibilities for how to use past game performances to craft a prior – the suggestions proposed here are meant to be a good baseline of possible priors that a coach could have about their goalies, but by no means do they comprise a complete list of good choices of prior.

Really Bad Starts (RBS)

Really Bad Starts, abbreviated RBS, is a statistic created by Rob Vollman (now with the Los Angeles Kings) for the Hockey Abstract. A goalie is credited with an RBS whenever they start a game and finish with a save percentage below 0.85 – after all, teams whose starting goalie posted an RBS won only 12.35% of the time between 2014-2015 and 2020-2021. If we restrict that to games where only one of the two starting goalies posted an RBS, the teams whose goalie posted an RBS have a win rate of only 6.89% between 2014-2015 and 2020-2021 (both of these win percentages were computed using the MoneyPuck dataset). It would therefore seem like one good way to decide if our goalie is having a bad game is by attempting to classify their game as an RBS (I briefly discussed the process of doing this in a Twitter thread during the 2021 playoff series between the Hurricanes and Predators). Given the current posterior distribution of \(\theta\) after observing \(x\) goals allowed on \(n\) shots, we then want to compute

\[ \Pr \left( \text{RBS} \right) = \Pr \left( \theta \leq 0.85 \right) = \int_{0}^{0.85} \pi \left( y \mid x, \alpha, \beta \right) \ d y \]

Luckily, this is not difficult to compute. However, once we have \(\Pr \left( \text{RBS} \right)\), we need to use it to make a decision. A reasonable idea would be to classify the start as an RBS (and thus pull the starter) if \(\Pr \left( \text{RBS} \right)\) is greater than some threshold \(t \in [0, 1]\) – but how should we choose \(t\)? In order to choose \(t\), we first need to decide what the penalty should be for both types of incorrect decision. That is, if \(\Pr \left( \text{RBS} \right) \geq t\), but the goalie isn’t actually having a bad game, how should we penalize that wrong decision? What about if \(\Pr \left( \text{RBS} \right) < t\), but the goalie is actually having a bad game? Assigning costs to these decisions is crucial in choosing a threshold \(t\), but there are many ways to assign costs. If we are strictly trying to maximize our chances of winning, we may decide that we’d rather incorrectly pull the goalie during a good game than incorrectly leave in the goalie during a bad game, in which case we’d choose a lower value of \(t\). If instead our job is in danger and we want to avoid making a catastrophically bad decision in the eyes of fans, media, and ownership, we might be willing to leave in the starter for a longer time, rather than making a quick judgment that could turn out incorrect, in which case we’d want \(t\) to be much closer to 1. If all we are concerned with is making the right decision as much as possible, then we’d probably want to choose \(t = 0.50\), or very close to 0.50. The important thing is that there are multiple ways to choose \(t\) so that our decision rule is tailored to how we want to make our goalie decisions.

Comparing Values of \(\theta\)

While classifying starts as good or bad based on the CDF of \(\theta\) for our starter is a simple computation, decision rules crafted in that way do not take into account our beliefs about the backup goalie. As mentioned previously, we would probably consider a Vezina-winning goalie and an aging veteran backup goalie to have very different prior distributions on their save probabilities. Therefore, while we might have good reason to believe that our Vezina-winning starter is having a bad game, it could be the case that our starter having a bad game is still better than our backup having an okay game (by their respective standards). Instead of classifying the starter’s game just based on their own value of \(\theta\), we could compute the probability that the starter’s save probability is lower than the backup’s save probability. With respect to our model, if we assume prior distributions

\[\begin{align} \theta_{S} & \sim \text{Beta} \left( \alpha_{S}, \beta_{S} \right) \\ \theta_{B} & \sim \text{Beta} \left( \alpha_{B}, \beta_{B} \right) \end{align}\]

for the starting goalie and backup goalie, respectively, and we observe the starter give up \(x\) goals on \(n\) shots, we would then have

\[ \theta_{S} \sim \text{Beta} \left( \alpha_{S} + n - x, \beta_{S} + x \right) \]

and then we could compute

\[ \Pr \left( \theta_{S} \leq \theta_{B} \right) = \int_{0}^{1} \int_{0}^{y} \pi_{S} \left( z \mid x \right) \pi_{B} \left( y \right) \ d z \ d y \]

Similarly to before, since we know the form of \(\pi_{S}(\cdot)\) and \(\pi_{B}(\cdot)\), this is not difficult to compute with the right software. We would again need to choose a threshold \(t\) such that we pull the starter if \(\Pr \left( \theta_{S} \leq \theta_{B} \right) \geq t\) and leave in the starter otherwise, but this can be done exactly the same way as detailed above. Of course, there are likely other reasonable ways to convert a beta distribution into a decision rule – these are only a few suggestions.

Other Considerations

Assume now that we have a decision rule \(d(\cdot)\) that takes as input the parameters defining the posterior distribution of \(\theta\) for the starting goalie (and possibly some other arguments as well), and returns a decision either to pull the starter or leave in the starter. How should we be using it? That is, at what points during the game should we be considering a goalie change, and how should we be using game-state information to help us with this decision? Of course, we can only make a goalie change during a stoppage in play, but we almost certainly shouldn’t be considering a goalie change at every stoppage. Conventionally, we would only consider pulling the starter after a goal against or after an intermission – after all, the starter making a save would only serve to improve our judgment of them, and no other information except for goals allowed is affecting our posterior beliefs about the goalie. As for switching goalies coming out of an intermission, we may want to pull our goalie after a goal, but if there is little time left on the clock, we may choose to just wait that time out and regroup during the intermission.

That said, consider the following situation: our goalie gives up 3 goals in the first period, but the team in front of the goalie holds the opposition to only 7 shots on goal, while generating 16 shots on goal and scoring none. We’re likely not very confident in our goalie, but the rest of the team is playing well. We don’t want to make a quick judgment, so we leave the starter in net. However, our team scores 2 quick goals to begin the second period, and all of a sudden it’s a close game again. The last thing we want is for a bad goalie to tank this comeback before we can finish it off, so maybe we now have more to lose if a bad game from our starter winds up costing us in the end. Should we pull the starter now that we seem to have a better chance of actually winning the game? In other words, should we consult \(d(\cdot)\) for a recommended decision after scoring a goal ourselves? At the very least, this (maybe uncommon) example might influence us to consult our decision rule in some less-conventional instances.

Additionally, we need to consider common situations where we might want to overrule the decision rule. We probably won’t ever want to pull the starter if our decision rule is telling us that they’re having a good game, so instead we’ll only consider situations where \(d(\cdot)\) returns a decision to pull the starter, but we might have reason to be more conservative. A simple example would be the case where the starter gives up a goal on their first shot faced. If we choose the maximum likelihood estimators \(\hat{\alpha}_{\text{MLE}}\) and \(\hat{\beta}_{\text{MLE}}\) computed previously as our prior, we have \(\Pr \left( \text{RBS} \right) \approx 0.295\) with just the initial prior distribution, but if the goalie allows 1 goal on 1 shot, we then have \(\Pr \left( \text{RBS} \right) \approx 0.485\). Depending on our choice of threshold \(t\), this could result in a suggested goalie switch extremely early in the game. The uncertainty in our judgment would still be extremely large, and we’d be risking a lot of earned respect and reputation if we wound up being wrong. For this reason, a good idea would be to have a minimum number of goals allowed before evaluating the starter. That said, this minimum shouldn’t be too large, as if we waited until the starter allowed 5 goals to evaluate whether they’re having a bad night, it would almost always be too late for pulling the starter to make any difference.

We would also probably like to consider the amount of time remaining in the game, both when there is a lot of time remaining and when the game is nearly over. We likely wouldn’t want to pull the starter 2 minutes into the game, even if they did give up a bad goal early. On the other hand, if we’re down 4-2 with 10 minutes left in the third period, and our starter allows a fifth goal, does it really matter if we make the right decision about their play? Is there any reward to pulling them this late? Furthermore, what if this is the first game of a back-to-back, and the backup on the bench is starting tomorrow night? In that situation, we might just want to punt on tonight’s game, let tomorrow’s starter rest, and hope we can salvage the second night.

In a similar vein, we also need to consider information beyond just our decision rule \(d(\cdot)\) and some game-state data. A goalie giving up 2 goals on 4 shots in 10 minutes looks bad on the box score, but if both goals came from high-danger scoring chances, we might be willing to give the starter some leniency before pulling them in favor of the backup. This is where expected goals could come in handy – while we may not be able to obtain shot probabilities in real time, we can always consult them after the fact as a supplement to the immediate decision we can obtain from our decision rule.

Again: the most important thing here is the process behind creating the decision rule \(d(\cdot)\) and then making actionable decisions, regardless of whether they agree with the output of \(d(\cdot)\). This process is not intended to erase the need for subjective judgment about the starting goalie – it is merely meant to supplement our decision-making process with a mathematical model that captures the volatility in the task we’re trying to accomplish, in order to hopefully help us make better-informed decisions that could help us win more hockey games.