Shot Quality

I think that in the case of the vast majority of hockey fans, there is a considerable amount of resistance to non-mainstream hockey statistics. Shot quality is no exception to this.

I'm not going to go into specifics in terms of what shot quality actually is. There are multiple articles at Alan Ryder's site that describe in detail what shot quality is and how it is measured. Furthermore, if you're reading this blog, there is a good chance that you're already acquainted with the idea of shot quality.

Rather, the purpose of this post is to explore the construct validity of shot quality, as well as its reliability. Because the figures for shot quality against (hereafter SQA) are more readily available than those for shot quality for, this post will deal with the former.

Firstly, the issue of validity. The biggest issue with shot quality, I think, is concern over the degree to which the shot quality figures actually mirror reality. Do teams with higher SQA (i.e. worse) really give up more dangerous shots against, or is it merely an artifact produced by arena bias or some other element of the measurement process? I think that the best way to answer this is to look at the relationship between team SQA and team save percentage. If teams with lower save percentages tend to have higher SQA, then that's fairly solid evidence that shot quality is measuring something genuine. The data:



For every single season, there is a negative correlation between SQA and save percentage at the team level, meaning that, on average, the teams that give up more dangerous shots against tend to have lower save percentages. Granted, not all of the correlations are statistically significant, but I think the fact that all of the correlation are in the same direction is suggestive of a reasonably strong relationship. Interestingly, the correlation is somewhat lower post-lockout than pre-lockout. I suspect that part of this might have to do with the fact that the figures for 2006-07 and 2007-08 are only for road games only, which was done in order to reduce the effect of arena bias. The save percentage figures I used for 2006-07 and 2007-08 are for away games only, so it can't be caused by my attempting to correlate road SQA against with overall save percentage. Rather, the cause probably relates to the fact that the sample size has been halved, thus reducing the resolution of the data.

But about it's reliability? Is there a correlation between SQA at Time A and SQA at Time B? The easiest way to determine this would be to look at it's split-half reliability at the team level, which would involve, for example, determining the correlation between a team's SQA against in odd numbered games and the corresponding figure for even numbered games. However, because I only have the figures for the entire season, I'm not able to do this. What I can do, though, is determine its inter-year correlation at the team level. Is SQA repeatable in this sense?




The data is fairly unequivocal. For each season pairing, there is a considerable correlation, and the degree of relationship is highly consistent across seasons. Indeed, the strength of the correlation is remarkably similar to the inter-year correlation for shots against over the same period. Admittedly, the pre-lockout values could be partly driven by arena bias, but that seems unlikely considering that the values after 2006-07 are for road shot quality only and the correlation is equally strong. Therefore, it can be said with some degree of confidence that a team's SQA for any given season is highly correlated with the values for both the preceding and following seasons, and that this relationship does not appear to be mediated by arena bias.

While shot quality is not likely to be embraced by mainstream hockey fans anytime soon, I feel that it has tremendous utility in that it quantifies one of the two components - the other being shots against - of team defense. Based on these findings, SQA can be said to be:

1. A real measurement of the average relative dangerousness of the shots allowed by any given team, as manifested in the lower team save percentages of teams that allow relatively more dangerous shots on average.

2. A reliable, enduring element of team defense, as manifested in the substantial inter-season correlation at the team level.

Save Percentage and Shots Against

Recently, at Hfboards and elsewhere, I've come across the assertion that, among NHL goaltenders, there is a positive relationship between save percentage and shots against. Some examples.

Naturally, I was skeptical of the claim , not least because no compelling evidence was ever offered in support of it. Moreover, there doesn't appear to be any reason why this should be so. Conceivably, there could be some degree of trade off between shot quality and shots against at the team level, in that teams that are better at preventing shots do so at the expense of allowing more quality ones (at least proportionately) and vice-versa. However, there doesn't appear to be much of a correlation between the two.






The chart shows the correlation between shots against and shot quality against (at the team level) in the right column with the corresponding season in the left column. In only two of the seasons (2002-03 and 2003-04) was there something of a relationship. What's interesting, however, is that the correlations are positive, meaning that, in those particular seasons, the teams that were better at preventing shots also tended to allow lower quality shots. There is absolutely no evidence of a trade off between shots against and shot quality.

Of course, it's possible that there could be a relationship between save percentage and shots against independently of shot quality. It's often been suggested that goaltenders play better when they're frequently tested and poorer when underworked. If this were true, a correlation between save percentage and shots against should emerge at the team level. But what does the data say?






As can be seen, there isn't much of a pattern between the two variables. In some years, the correlation is positive; in other years, negative. What's important, I think, is that all the values are fairly close to zero. The only year where the correlation is even remotely significant at the 5% level is 1999-00, but the direction is contrary to what would be expected . The important point is that teams that give up more shots against do not tend to have higher save percentages.

There is absolutely no evidence that high shot totals have an inflationary effect on goaltender save percentage.

Why, then, is it often argued that such a relationship exist? More than anything else, the phenomenon seems to be driven by wishful thinking on the part of the claimants, a disproportionate number of whom belong to certain fan bases. I'll say no more.