Using Win Probabilities to Optimize Goalie-Switching Decisions

October 4, 2018: Washington 6 at Pittsburgh 7 (OT)

For over 90% of this game, both teams would have been better off pulling their respective starters, but somehow both Matt Murray and Braden Holtby were allowed to play the entire game. At one point during the game, Pittsburgh stood to gain an estimated 21.2% win probability by pulling Murray, and at a different point during the game, Washington stood to gain an estimated 19.3% win probability by pulling Holtby.

January 12, 2019: NY Rangers 2 at NY Islanders 1

On the opposite extreme, this game featured two goalies having fairly strong games, so it shouldn’t be surprising that both teams would stand to lose a significant portion of their respective win probabilities by pulling their respective starters.

April 6, 2019: Vegas 2 at Los Angeles 5

As for a more typical NHL game, early on both goalies start out strong, but after a few first period goals by Los Angeles, we can see that the home team’s win probability would decrease if the away team switched goalies (indicating that the away team’s win probability would increase if they switched goalies), while for most of the game the home team’s starter is giving them a far better chance to win than their backup would.

Florida Panthers vs. Washington Capitals, Game 5

With the series tied at 2 games apiece, this was a crucial game for both teams. However, for the beginning of the game, the Panthers didn’t seem to realize this, as they conceded a 3-0 lead to the Capitals less than 4 minutes into the second period, before eventually mounting a comeback and winning 5-3 to take a 3-2 series lead. While a 5-3 score may seem fairly standard, this game provides a good example of a rare occurrence where going against traditional thinking may be best when making goalie-pull decisions. The model output for this game is shown below. Note that win probabilities are expressed as probabilities between 0 and 1, so to get the associated percentage values simply multiply them by 100. Following the conventions from my first post, in the columns representing estimated win probability changes, a positive value (highlighted in yellow) means the team would be better off switching goalies, and a negative value (highlighted in purple) means the team would be better off sticking with their current goalie. Gray rows represent evaluations performed during intermissions.

When the Capitals scored to make it 3-0, the decision machine recommended that the Panthers switch to Spencer Knight in net, with an estimated win probability gain of 1.9% associated with the switch. However, the Panthers didn’t switch to Knight, and instead began their comeback, which actually resulted in an increase in the estimated win probability gain from switching to Knight. This may not be intuitive, but think of it this way – when down 3-0, Knight was the better choice (according to this model), but the estimated win probability gain was so minimal because even with a better goalie, the Panthers are still down by 3 goals. As they begin to close the gap on the scoreboard, however, Knight is still the better choice, and the fact that the game is now tighter means that Knight provides a bigger boost to the Panthers’ win probability. In a way, the Panthers have more to lose by sticking with Bobrovsky during this comeback. Nevertheless, the Panthers never switched to Knight in this game, and they did eventually complete the comeback. However, this is a great example of a game where I believe a coach should be thinking about a goalie change after their team scores a goal. In reality, no coach would ever consider pulling a goalie under these circumstances (they would typically only do so after a goal against or after an intermission), but the output above should ideally get coaches to think differently about when to judge their goalies.

Note also that the decision machine recommends Vanecek over Samsonov at the time of the Capitals’ first goal, despite the fact that Samsonov had not allowed any goals to that point – this is because the priors on the four goalies’ save probabilities are estimated using their regular-season starts, and Vanecek’s regular-season numbers were much better than Samsonov’s numbers. This serves as another reminder that these numbers without the proper context are not particularly useful, and ideally a coach would weigh the output of this model along with their subjective opinion about how their team’s two goalies have performed in the current game and in previous games.

Calgary Flames vs. Edmonton Oilers, Game 1

The “Battle of Alberta” got off to a great start, with the Flames taking Game 1 9-6 over the Oilers. With that many goals, it shouldn’t come as a surprise that the game featured some bad goaltending and some interesting decisions by both coaches. The model output for this game is shown below.

One of the first things to note about this game is that Mike Smith was pulled extremely early. Calgary was up 2-0 by the end of the first minute of the game, with Smith giving up a goal on his first shot faced and 2 goals on his first 3 shots faced. While he managed to keep the puck out of the net for a few more minutes, he gave up his third goal on his tenth shot faced only 6 minutes into the game, prompting Jay Woodcroft to pull him for Mikko Koskinen. As shown in the table above, Edmonton could have gained 9.1% by pulling Smith after Calgary’s first goal or 11.1% by pulling Smith after Calgary’s second goal, but instead chose to leave Smith in the game. While Smith is notorious for being an extremely high-variance goalie, it’s hard to fault Woodcroft for not pulling Smith after only one or two goals, even with the large estimated win probability gains from this model. Nevertheless, Edmonton did eventually gain 5.5% in win probability by pulling Smith for Koskinen, so Woodcroft deserves some credit for making a change so early in the game. Also of interest regarding Edmonton’s goalies is that despite Koskinen giving up 5 goals by the end of the game (the ninth Calgary goal was an empty-net goal), at no point would Edmonton have been better off by going back to Smith. Despite Koskinen also having a relatively poor performance, he was still a better option than Smith. All things considered, I think Woodcroft made some smart decisions about his goalies in a game where his team lost 9-6.

On the other hand, Darryl Sutter may have come away from this game with a win, but his process was not very sound according to this model. Despite the model suggesting that Jacob Markstrom should have been pulled after allowing 1 goal on 5 shots and after allowing 2 goals on 9 shots, Calgary was up 3-1 and 5-2 at both of those times, so again it’s hard to fault Sutter for sticking it out with Markstrom. However, when Edmonton began their comeback at the end of the second period, with Markstrom allowing his third, fourth, and fifth goals on only his 12th, 14th, and 17th shots faced, respectively, Sutter may have wanted to consider pulling Markstrom after any of those goals. In my subjective opinion, this is one of the best instances in which this model is particularly useful – your goalie is having a terrible game, and yet not only are you still in the game, but you have the lead! Sure, it’s the playoffs and Markstrom is the guy who got you to this point, but he is clearly not having a good game by his standards. At the very least, Sutter probably should have been strongly considering going to Vladar to start the third period (which would have provided an estimated win probability gain of 11.9%). However, Markstrom came out to start the third period, and less than two minutes into the period, Markstrom allowed his sixth goal of the night on his 20th shot faced, and just like that Calgary’s four-goal lead had been erased. The estimated win probability gain from going to Dan Vladar was up to 13.2%, which would have bumped Calgary’s then-current win probability of 38.7% over 50%. Yet somehow, Calgary and Markstrom managed to stop the Edmonton surge, breaking the tie shortly after and going on to win the game. To me, this feels like bad process leading to good results for Sutter and the Flames, but part of what makes the NHL playoffs so exciting is that only the end result matters. That said, Calgary did go on to lose the next four games, with Markstrom playing poorly in most of those games and Sutter never giving Vladar the net in relief. It’s unfair for me to suggest that Sutter’s refusal to pull Markstrom in Game 1 foretold bad things for Calgary (since one of the critical assumptions underlying this model is that all games are independent), but it seems as though Sutter’s conservative decision-making in Game 1 was a recurring habit through the remainder of the series that inevitably contributed to Calgary’s early elimination.

Colorado Avalanche vs. Edmonton Oilers, Game 1

In the first game of their next series, Edmonton again found themselves in a shootout that was somehow only the 2nd-highest scoring game of the 2022 playoffs to that point (with their previous Game 1 having one more goal), eventually losing to Colorado by a score of 8-6.

Much like Game 1 of the Battle of Alberta, both teams faced some interesting decisions in this game. Edmonton opened the scoring early, scoring on only their second shot against Darcy Kuemper. However, Colorado answered back shortly after, on only their second shot against Mike Smith. After this point, both teams would have been better off switching goalies at every evaluation point until after Colorado’s sixth goal, when Edmonton finally pulled the trigger and pulled Smith in favor of Koskinen, and Pavel Francouz was forced into action because of an injury to Kuemper. While Smith allowing 6 goals before the end of the second period certainly looks bad, the Oilers were getting heavily outplayed by Colorado, so I don’t fault Jay Woodcroft too much for not pulling Smith early in the game. That said, even when you don’t think your goalie is to blame for most of the goals he’s allowed, it’s hard not to pull him when you’ve given up 5 goals on 22 shots in Game 1 of the Conference Finals. All things considered, I think this is a situation where the decision machine output doesn’t line up very well with a subjective opinion of Smith for the first 3 or 4 goals, but at some point a pull should become the obvious decision, even if it’s only to try and reverse the momentum of the game.

On the other hand, Jared Bednar can also be forgiven for not pulling Darcy Kuemper, even though he certainly didn’t play well before his injury forced him out of the game. Kuemper allowed 1 goal on 2 shots, 2 goals on 8 shots, and 3 goals on 13 shots before eventually being removed from the game because of injury. In the majority of hockey games where your goalie performs this poorly, there’s no way he should be allowed to continue in the game, but luckily for Bednar and the Avalanche, their offense was firing on all cylinders and Kuemper’s poor performance never really put them in jeopardy. Francouz did allow 3 goals in relief of Kuemper, but by that point, the game was nearly over, and even with the decision machine not being aware of Kuemper being unavailable, the Avalanche were better off with Francouz anyway. Not only did Francouz outperform Kuemper in this game, but he also won the next 3 games in a row to sweep Edmonton en route to the Stanley Cup Final while Kuemper was out with injury.