Tuesday, April 27, 2010

The Repeatability of Special Teams Performance

In my post on playoff probabilities, one of the methods in which I calculated each team's expected winning percentage was on the basis of the underlying numbers.

Under this model, shot volume on the powerplay, shot prevention on the penalty kill, as well as penalty differential, were incorporated as determinants of special teams goal differential. However, neither shooting percentage on the powerplay nor save percentage on the penalty kill were used as predictors.

Initially, my intention was to include both variables within the model. However, after looking at the relationship between team powerplay shooting percentage in even numbered games and team powerplay shooting percentage in odd numbered games in the 09-10 regular season, I discovered that there was essentially no correlation. I then did the same thing for the 07-08 season, and the result was the same: no relationship.

I found this to be unusual, given that I had looked at the distribution of powerplay shooting percentage in the past and found that the team-to-team spread was somewhat broader than what one would expect if there was no skill component. Nevertheless, my exercise had revealed the absence of any split-half correlation, thus necessitating the exclusion of PP S% from the model.

(As mentioned above, I also excluded PK save percentage, even though I had not specifically examined its repeatability. This was somewhat unjustified given that, as discussed below, team PK SV% is somewhat repeatable. However, the regression is fairly strong and, even though I ought to have taken it into account, it's exclusion didn't affect things too greatly.)

In any event, my curious findings prompted the following question: To what degree is special teams performance repeatable?

Real Effects

Vic Ferrari had an excellent post about a year ago where he looked at the various components of team even strength performance -- specifically, shooting percentage, save percentage, and shot differential -- and determined the extent to which each component was repeatable. Specifically, his method involved looking at each team's shooting percentage, save percentage, and shot differential, all at EV with the score tied, in 38 randomly selected games from the 2008-09 season. He then looked the same variables over a separate 38 game sample, and determined the correlation between the two sets of games. The exercise was then repeated over 1000 simulations.

The rationale behind the exercise is a simple one -- as expressed by Vic, "if an element of nature is affected by something other than randomness, that it should sustain itself from one independent sample to another." Thus, if the observed correlation is significantly non-zero, it can be assumed that the variable is at least partly determined by factors other than luck. On the other hand, if the observed correlation is insignificantly different than zero, then fluctuations in the variable are assumed to be primarily luck driven.

I decided to apply a similar technique in order to determine the degree to which the components of special teams performance are governed by 'real effects.' Specifically, my methodology involved the following:
  • I obtained special teams data at the team level for each season from 2003-04 to 2009-10
  • Within each season, I looked at team performance on specials teams at the level of individual games
  • In particular, I looked at the following variables: powerplay shooting percentage, penalty kill save percentage, powerplay shot rate (shots for divided by time on ice), penalty kill shot rate (shots against divided by time on ice), and powerplay ratio (the ratio of powerplays drawn to powerplays conceded)
  • However, shooting rates were not examined for 2007-08, 2008-09, and 2009-10, as I was not able to obtain data on PP TOI and PK TOI for those seasons
  • Empty net goals were excluded when calculating shots and goals
  • For each team, I randomly selected 20 home games and 20 road games, combined the two sets of games, and looked at how that team performed within that sample with respect to the above stated variables
  • I then did the same thing for 40 other randomly selected games (again, consisting of 20 homes and 20 road games)
  • I then looked at the correlation between the two sets of games for each of the listed variables
  • I repeated the exercise 1000 times, for each of the six seasons
The results


I should note that the final highlighted column shows the averaged value for each variable.

As indicated by the table, both generating shots on the powerplay and preventing shots on the penalty kill appear to be largely ability driven measures. The same applies to drawing more powerplays than the opposition.

Not surprisingly, both PP S% and PK SV% are less ability driven than the other three variables. It's worth noting that PK SV% appears to be more reliable than PP S%. I presume that this can be attributed to the influence of the goaltender on PK SV%.

Wednesday, April 14, 2010

Corrected Playoff Probabilities

Because I forgot to include EV goals when calculating each team's corsi with the score tied, I decided to re-run the UNDERLYING #'s simulation using the corrected probabilities.

The results aren't too different.

Expected Winning Percentage by Team

In response to a question raised in the comments to my post on playoff probabilities, I figured that it would be useful if I posted each team's expected winning percentage according to the two described methods.

The teams are ranked according to pythagorean winning percentage.


EDIT:

I've altered the chart so as to include even strength goals in the calculation of Corsi with the score tied. The values don't change all that much -- in fact, hardly at all, but I figured that I'd post it if only for accuracy's sake.

Tuesday, April 13, 2010

Playoff Predictions

S.J - COL

The deceptive nature of Colorado’s success this season has been well documented by some of the more statistically inclined members of the hockey blogosphere. As I presume that those reading are familiar with that fact, I won't go into any detail. The Sharks are the better team in virtually every facet of the game, save for perhaps goaltending. As with any series, an upset is always possible, but I think that a lot would have to go wrong for San Jose to lose.

S.J in 5.

CHI –NSH

There isn’t really a lot to be said about this series. I don’t think that the Predators are a bad team, but they have the worst goal differential among the playoff teams in the West. Chicago, on the other hand, is probably the best team in the entire league. They have the best goal ratio once schedule difficulty and empty netters are taken into account, and they were far and away the most dominant team in the league in terms of outshooting. I expect them to advance without too much difficulty.

CHI in 5.

VAN-L.A

I don’t find Vancouver to be all that impressive, but I think that they’re the right pick here. They have the better goal differential, the better shot ratio at EV with the score tied, the better goaltender, and they’ll be starting the series at home. The Kings are respectable and while I suspect that I’ll end up cheering for them here, I just can’t justify picking them in the result.

VAN in 7.

PHX-DET

I’ve taken quite a liking to Phoenix ever since I watched them smoke the Kings on the first Saturday of the season. Imagine my disappointment when I found out that the Red Wings finished 5th. Outside of Chicago, I don’t think that the Coyotes could have asked for a less favourable draw in the first round.

I’ve remarked in the past about how Phoenix has been one of the stronger teams in the league terms of outshooting at EV this year, which is impressive given where they were last season. However, Detroit’s numbers are even better in that respect. Additionally, Detroit’s underlying numbers have improved over the course of the season, whereas the reverse has been true for Phoenix. I’m not sure if that’s terribly relevant to each team’s chances, although it can’t be a good thing from the Coyotes’ perspective.

Finally, the Wings are clearly the better squad on special teams. There isn’t much of a difference between the two teams in terms of shot prevention on the penalty kill, but the Wings are much better at generating shots – not to mention goals – on the PP.

All in all, while I think that the Coyotes are largely legit, they’re clearly overmatched here.

DET in 6.

WSH – MTL

I don’t think that Washington is as strong as its goal differential would imply. However, even if that’s accounted for, they’re still the better team by a substantial margin. Neither club appears to have much of an advantage over the other on special teams, but the Habs get bombed in terms of shots at even strength whereas the Capitals are above average in that respect. The Capitals should dominate the play at even strength and, unless Halak can bail his team out, that should be the difference.

WSH in 5.

N.J – PHI

I agree with Sunny Mehta -- these two teams are reasonably close to one another in terms of ability, but the Devils have a clear advantage in goal. Whereas Brian Boucher has a career even strength save percentage of 0.910, the corresponding figure for Brodeur is 0.922. The true difference in ability is probably larger if one considers that the shot recorder in New Jersey undercounts and that Brodeur has generally been better post-lockout than pre-lockout. Ordinarily I try to refrain from basing a pick on goaltending alone, but when the teams are relatively evenly matched and the gap in goaltender ability is large, I think that it’s reasonable to do so.

N.J in 7.

BUF – BOS

Although the Sabres may have the better record and goal differential, the Bruins strike me as the better team here. The two teams exhibit similar profiles on special teams (good PK, poor PP), but Boston appear to be the better team at even strength. The Bruins were second to only Chicago in terms of outshooting at EV with the score tied, whereas the Sabres were around the league average in this regard. The Sabres actually had the better EV goal differential, but only by virtue of the percentages. I suspect that Boston’s territorial dominance will prevail as the percentages equalize from this point forward.

Some may argue that the Sabres have the better goaltender in Miller, but I’m not sure if that’s necessarily true. Miller finished the season at 0.928 at EV and 0.919 on the PK. His career values are 0.922 and 0.880, respectively. I think it’s reasonable to assume that his career values are more reflective of his ability than this season’s numbers. To the extent that Buffalo has the better goaltending, the difference probably isn’t large.

BOS in 6.

PIT – OTT

This matchup strikes me as the Eastern Conference analog of the Vancouver-LA series. I don’t think that the two teams are that far apart in terms of quality, but the Penguins have the advantage in pretty much every conceivable area that relates to winning – goal differential, outshooting (both in general and at EV), penalty differential, special teams and, as with the Canucks, the higher seed. Although I don’t necessarily think that the Senators will get blown out of the water, there’s simply no rational basis for picking them.

PIT in 6.

Playoff Probabilities

In order to get a sense of each team's chances, I decided to run a couple simulations of the first round.

For the first set of simulations, I calculated each team's winning percentage on the basis of pythagorean expectation after correcting for schedule difficulty, empty netters and shootout goals. In the charts displayed down below, the probabilities determined on this basis can be found in top half of each individual chart (next to the cell titled 'PYTHAGOREAN').

In the second set of simulations, the methodology was somewhat more complicated. Without getting too specific, I computed each team's theoretical winning percentage on the basis of the following inputs:
  • Each team's corsi ratio with the score tied during the regular season (as a determinant of shots for and against at EV)
  • The career EV save percentage of each team's starting goalie (as a determinant of each team's EV save percentage and the shooting percentage of its opponent). Each goalie's career save percentage was regressed to the league average based on the number of career EV shots faced to date (for goalies facing fewer shots, the regression was stronger; for goalies facing more shots, the regression was weaker)
  • Each team's tendency to draw and surrender powerplays during the regular season (as a determinant of time spent on the powerplay and penalty kill)
  • Each team's shot rate on the powerplay and shot rate against on the penalty kill during the regular season (as a determinant of powerplay goals for and against)
The probabilities associated with these inputs can be found in the bottom half of each individual chart (next to the cell titled 'UNDERLYING #'s').

For each set of simulations, I simulated the first round 10 000 times. Home advantage was taken into account for both sets of simulations. The results are displayed below, with the Eastern Conference following the West.




The row next to each of the four numbers shows each team's probability of winning the series in that many games. The highlighted row shows each team's chance of winning the series.

By way of example, consider the San Jose Colorado series. If each team's winning percentage is computed on the basis of pythagorean expectation, the Sharks have a 11.3% chance of winning the series in a sweep and a 68.5% of winning the series.

If, on the other hand, the second method is applied, the Sharks have a 15.5% of winning in a sweep and a 77.9% chance of winning overall.

Overall, the two methods yield comparable results, except in the case of the DET-PHX and BUF-BOS matchups. The first method suggests that the Coyotes should win slightly over half the time, whereas the second indicates that the Wings are the clear favorite.

The discrepancy is even greater for the Sabres and Bruins matchup. According to the PYTHAGOREAN method, the Sabres should win some two-thirds of the time, yet the second method produces the opposite result.


Friday, April 9, 2010

The Leafs, Scoring Chances and Corsi

I recently got around to updating my database for the 09-10 season and, in looking over the EV stats for each team, I noticed that the Leafs continue to have one of the best corsi ratios in the league at EV with the score tied.



I think that this is unusual for a couple reasons.

For one, the 08-09 Leafs were a poor team according to this metric. While last year's squad outshot the opposition in a general sense, their corsi ratio with the score tied was 0.94, good for 21st in the league. Thus, if one considers corsi ratio with the score tied to be a crude measure of a team's ability at even strength, the Leafs would appear to be one of the most improved teams in the NHL (looking strictly at EV play, of course).

Secondly, despite soundly outshooting the opposition, the Leafs have one of the worst EV goal differentials in the league. I haven't filtered out empty netters yet, but only the Lightning, Oilers, and Jackets are worse than Toronto in terms of goal differential at even strength. This despite directing some 500 more shots towards the other team's net at EV than their opponent over the course of the season. The effect isn't as extreme when the score is tied -- they're only -7 -- but the unusual profile remains.

The tendency to outshoot without outscoring has led some to question whether the Leafs do, in fact, outplay the opposition at even strength, or whether the shot numbers are deceiving.

One way to settle the issue is to look at the Leafs scoring chance numbers. If Toronto's scoring chance ratio broadly parallels its shot ratio at EV, then that ought to dispel notions that the Leafs don't legitimately outplay the opposition, or that they shoot from everywhere.

Slava Duris, whose blog can be found here, has been recording scoring chances for Toronto over the course of the season. To date, he's posted 53 of the games for which he's recorded chances.
Taking those 53 games in particular, I looked at how many even strength scoring chances the Leafs had with the score tied, and how many their opponents had. I then determined how many shots the Leafs directed towards the opposition's net -- again, only at EV with the score tied -- in those same 53 games, and did the same for their opponents. The raw data can be viewed below.



Overall, there were 474 even strength scoring chances with the score tied in the 53 games sampled. Of those 474 chances, Leafs generated 252, whereas the opposition generated the remaining 222. Thus, the Leafs scoring chance ratio with the score tied was 1.14.

In terms of corsi with the score tied, the Leafs directed 1008 shots towards the opposition's goal, and had 915 directed toward their own, thus giving them a corsi ratio of 1.10.

In other words, the Leafs actually did better in terms of scoring chances than in terms of corsi over the 53 games examined.

Granted, this doesn't allow one to conclude that the Leafs are a better team than their corsi ratio would suggest. For example, if we assume that Toronto's underlying scoring chance ratio is identical to its corsi ratio (1.10), then the probability of it generating at least 252 chances out of 474 randomly selected chances is 0.376 (or, if one prefers, the probability of it having a corsi ratio at least as good as 1.14 in a sample of 474 chances). In other words, the two values are not significantly different from each other.

Nevertheless, it would appear that the Leafs have managed to outplay the opposition at even strength over the course of the season, their rotten goal differential notwithstanding.

Tuesday, March 9, 2010

Shot Recording Bias: Part n

This post is the third post that I've made on the subject. For a more detailed discussion of the methodology and reasoning applied, please refer to my first two posts ([1] [2]).

After looking over my previous post on the subject -- the one examining Florida and New Jersey specifically, I realized that I'd made an error in inputting the data for the 2007-08 and 2008-09 seasons. Here are the corrected charts. It may be necessary to enlarge them in order to properly view the information.

New Jersey


In my original post, I concluded that the shot recorder in New Jersey undercounts shots on goal. The corrected data does nothing but affirm that conclusion.

The only major difference is that the chart contained in my original post incorrectly showed that there were more shots counted in New Jersey home games than New Jersey road games in both 2007-08 and 2008-09. This led me to suspect that the bias may no longer persist, notwithstanding the fact that the shooting percentage in New Jersey home games was higher than the shooting percentage in New Jersey road games during the two seasons in question.

However, as is evident from the corrected chart, there were actually fewer shots in Devils home games for both 2007-08 and 2008-09. This is consistent with the data from previous seasons, the corresponding shooting percentage data for 2007-08 and 2008-09, as well as the undercounting hypothesis.

Florida



The corrected data for Florida, however, does serve to affect my conclusions somewhat. While the home-road shot gap for 2007-08 and 2008-09 is similar in magnitude to that observed in the previous three seasons, the shooting percentage data for those two seasons suggests that an overcounting bias may have emerged. However, I'm reluctant to assert the existence of a bias on the basis of two seasons worth of data alone, especially considering that the shot gap has not increased materially.

Other Arena Recording Biases

Given that we're on the subject, I figured I'd take this opportunity to explore the issue of shot recording bias more generally.






The above tables show each team's home-road splits for shots on goal and shooting percentage from 2003-04 to 2008-09. The first table shows the 15 teams that had the greatest number of recorded shots on goal in home games relative to road games, and ranks those teams in descending order. The second table basically shows the reverse.

The first highlighted column in each table displays the number of shots recorded in home games, minus the number of shots recorded in road games.

The second highlighted column displays road game shooting percentage minus home game shooting percentage.

Where there exists a significant positive value in both columns, an overrecording bias is implied.

Conversely, where both values are significantly negative, an underrecording bias is implied.

Looking at the two tables together, it would appear that the shot recorders in Colorado, Ottawa, Nashville and Boston overcount shots to some degree, whereas the recorders in Minnesota, Dallas, St. Louis and Vancouver are seemingly guilty of undercounting.

Of course, a more rigorous analysis is required before any conclusions can be reached.

Colorado


Ottawa


Nashville

Boston

Minnesota

Dallas

St.Louis

Vancouver


The above tables break down the home-road shot and shooting percentage splits by game state and season for the eight listed teams. I'm not sure if these tables add all that much on top of the aggregated data presented earlier, although I think that their inclusion is valuable for two reasons.

For one, the home games of some teams might have featured more special teams play over the period in question, even by sheer chance alone. As both shot rate and shooting percentage increase significantly on special teams relative to even strength, this factor can potentially distort the overall data.

Secondly, it's important to break down the data by season in order to see if any of the apparent recording biases are time-limited -- that is, present in some seasons but not others. For example, it's conceivable that some teams have employed more than one arena statistician at different points over the last seven years.

As for the tables themselves, one thing that strikes me as unusual is that the home-road shooting percentage gap for the Wild is quite large on special teams yet virtually non-existent at even strength (indeed, not even in the predicted direction). I can't think of any reason why this would be so, although it leads me suspect that there may be no bias. The home-road shot gap is large, but that could be a product of the Wild playing more conservatively at home.

Looking at the data collectively, there's overwhelming evidence of a recording bias in Dallas and Colorado, strong evidence of one in Vancouver and Ottawa, and moderate evidence of bias in the other four locations.


The above table requires some description. It essentially shows the 95% and 99% confidence intervals for each team's home shooting percentage (that is, the shooting percentage by both teams) during the period in question (2003-04 to 2008-09). The intervals were generated by assuming that each team had the same underlying shooting percentage on the road as at home, and that shots were recorded accurately irrespective of game location.

The final column shows what the shooting percentage in each team's home games actually was over that timeframe. Values colored light blue fall outside the 95% confidence interval. Highlighted values fall outside both confidence intervals. White colored values are within both confidence intervals.

A specific example will be illustrative. The Stars shot 0.081 at EV from 2003-04 to 2008-09. Using that value as their underlying home shooting percentage, their home shooting percentage would be expected to fall within 0.075 and 0.087 95% of the time, and between 0.073 and 0.089 99% of the time. The observed value was 0.089, which strongly implies that shots were undercounted in Dallas during this period.


Of course, assuming that each team's actual road shooting percentage is roughly equivalent to its underlying road shooting percentage is somewhat questionable. For example, if the underlying shooting percentage in a team's road games is 0.092, the 95% confidence interval after 12000 shots -- the average number of shots in road games for teams during the 5 year period -- is roughly between 0.087 and 0.098.

That being the case, the above table represents a slightly different approach. The left-hand column titled 'DIFF' shows the absolute difference in home and road shooting percentage for each team over the entire sample, for both EV and overall. The right-hand column titled 'PROB' displays the probability of a difference that large or larger occurring by change alone (over 100 simulations).

So, by way of example, the difference between the EV shooting percentage in Dallas road games and Dallas home games was 0.008. The probability of a difference at least that large arising from chance alone is 5%. In other words, it probably isn't the result of chance, but, rather, because shots have been undercounted in Dallas over that period.

Conclusions

So, what can we conclude from all that?
  • The shot recorder in New Jersey undercounts (this was addressed in a previous post)
  • The shot recorder in Dallas undercounts
  • The shot recorder in Colorado overcounts
  • The shot recorder in Vancouver almost certainly undercounts
  • The shot recorder in Ottawa probably overcounts
  • The shot recorders in Boston and Nashville may overcount, but the evidence is not conclusive
  • The shot recorders in St. Louis and Minnesota may undercount, but the evidence is not conclusive