I'm not going to go into specifics in terms of what shot quality actually is. There are multiple articles at Alan Ryder's site that describe in detail what shot quality is and how it is measured. Furthermore, if you're reading this blog, there is a good chance that you're already acquainted with the idea of shot quality.
Rather, the purpose of this post is to explore the construct validity of shot quality, as well as its reliability. Because the figures for shot quality against (hereafter SQA) are more readily available than those for shot quality for, this post will deal with the former.
Firstly, the issue of validity. The biggest issue with shot quality, I think, is concern over the degree to which the shot quality figures actually mirror reality. Do teams with higher SQA (i.e. worse) really give up more dangerous shots against, or is it merely an artifact produced by arena bias or some other element of the measurement process? I think that the best way to answer this is to look at the relationship between team SQA and team save percentage. If teams with lower save percentages tend to have higher SQA, then that's fairly solid evidence that shot quality is measuring something genuine. The data:
For every single season, there is a negative correlation between SQA and save percentage at the team level, meaning that, on average, the teams that give up more dangerous shots against tend to have lower save percentages. Granted, not all of the correlations are statistically significant, but I think the fact that all of the correlation are in the same direction is suggestive of a reasonably strong relationship. Interestingly, the correlation is somewhat lower post-lockout than pre-lockout. I suspect that part of this might have to do with the fact that the figures for 2006-07 and 2007-08 are only for road games only, which was done in order to reduce the effect of arena bias. The save percentage figures I used for 2006-07 and 2007-08 are for away games only, so it can't be caused by my attempting to correlate road SQA against with overall save percentage. Rather, the cause probably relates to the fact that the sample size has been halved, thus reducing the resolution of the data.
But about it's reliability? Is there a correlation between SQA at Time A and SQA at Time B? The easiest way to determine this would be to look at it's split-half reliability at the team level, which would involve, for example, determining the correlation between a team's SQA against in odd numbered games and the corresponding figure for even numbered games. However, because I only have the figures for the entire season, I'm not able to do this. What I can do, though, is determine its inter-year correlation at the team level. Is SQA repeatable in this sense?
The data is fairly unequivocal. For each season pairing, there is a considerable correlation, and the degree of relationship is highly consistent across seasons. Indeed, the strength of the correlation is remarkably similar to the inter-year correlation for shots against over the same period. Admittedly, the pre-lockout values could be partly driven by arena bias, but that seems unlikely considering that the values after 2006-07 are for road shot quality only and the correlation is equally strong. Therefore, it can be said with some degree of confidence that a team's SQA for any given season is highly correlated with the values for both the preceding and following seasons, and that this relationship does not appear to be mediated by arena bias.
While shot quality is not likely to be embraced by mainstream hockey fans anytime soon, I feel that it has tremendous utility in that it quantifies one of the two components - the other being shots against - of team defense. Based on these findings, SQA can be said to be:
1. A real measurement of the average relative dangerousness of the shots allowed by any given team, as manifested in the lower team save percentages of teams that allow relatively more dangerous shots on average.
2. A reliable, enduring element of team defense, as manifested in the substantial inter-season correlation at the team level.
No comments:
Post a Comment