tag:blogger.com,1999:blog-3299311926633621468.post7217193210695066544..comments2015-03-01T22:26:00.128-08:00Comments on Objective NHL: Loose Ends - Part I: Predicting Future SuccessJLikenshttp://www.blogger.com/profile/02570453428274983835noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-3299311926633621468.post-45951071888071044282015-01-04T11:42:17.143-08:002015-01-04T11:42:17.143-08:00Hey. I just want to add my 2 cents about sample s...Hey. I just want to add my 2 cents about sample size and prediction quality.<br /><br />-Smaller training set favors Corsi over W% and GR<br />-Larger training set reduces the differences among metrics<br />-Smaller test set reduces the accuracy of all predictions<br /><br />Taking these observations together, I think averaging out randomness is the most important factor here. Corsi data has more hidden samples than goals or wins. Games occur on a time-scale of days, goals on a time-scale of minutes, and possession-changes on a time-scale of seconds. So you can say "I averaged 20 Corsi values and 20 GR values", but the values you started with were already averages.<br /><br />As you increase the size of the training set, you increase the number of hidden samples for all of the metrics, which seems to reduce the differences among them.<br /><br />So if you want to know which metric is "best", it depends on your situation.<br /><br />If you need to train your prediction on a small number of games, because a season just started, or a line just changed, then go with Corsi.<br /><br />If you have an entire season of data to work with, then it would seem that GR and W% are just as good or maybe better.<br /><br />I think you could get a better answer with an n-fold cross-validation. That means you select the training and test sets randomly many times and average the results. This lets you use a small test-set size and still get reliable results. For instance, you could train on 75 games and test of 5. Then reselect the training and test sets 10 times and average the results. I suspect you will see W% and/or GR surpass Corsi at some point.<br /><br />If I'm right, it would mean that wins and goals are actually better predictors of future success than possession, but it takes a long time to get enough data to use them.Alanhttp://www.blogger.com/profile/13336579457813518267noreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-5323411988189566932013-05-03T12:54:34.241-07:002013-05-03T12:54:34.241-07:00I have to admit that I really like to make predict...I have to admit that I really like to make predictions and the best thing about it. It is to be accurate. Hostpph.comhttp://www.hostpph.comnoreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-39077852513689286702011-03-19T13:12:43.090-07:002011-03-19T13:12:43.090-07:00Michael:
I see what you're saying.
You'r...Michael:<br /><br />I see what you're saying.<br /><br />You're absolutely correct that future results become more random as the sample size decreases. <br /><br />However, this can't explain how the predictive power of the variables changes relative to each one another.<br /><br />This can be illustrated by randomly selecting 20 games for each team, looking at each of three variables over that sample, and then determining how each variable predicts future winning over a second, independently selected 20 game sample.<br /><br />We can then compare those results to how well each of the variables predicted future results in the "60=>20" grouping.<br /><br />If EV GD and W% <b>do not</b> become better predictors of future winning (relative to EV Corsi Tied) as the sample size increases, then the results in the "20=>20" grouping should roughly match those in the "60=>20" grouping. <br /><br />If, however, EV GD and W% <b>do </b> become better predictors (again, relative to EV Corsi Tied) as sample size increases, then EV Corsi Tied should have more predictor power, relative to the other two variables, in the "20=>20" grouping than in the "60=>20" grouping.<br /><br />Using the 2009-10 season as our sample, I got the following:<br /><br />60=>20 (from table in post)<br /><br />W%: 0.366<br />GR: 0.358<br />Corsi T: 0.346<br /><br />20=>20 <br /><br />W%: 0.147<br />GR: 0.177<br />Corsi T: 0.275<br /><br />Corsi Tied has much more relative predictive power in the second grouping than in the first, indicating that it's not just a matter of results becoming more variable as the sample size decreases. Rather, goal ratio and winning percentage become relatively stronger predictors as the season moves forward and the amount of information we have about each team grows.<br /> JLikenshttp://www.blogger.com/profile/02570453428274983835noreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-43065808654708785272011-03-19T12:32:36.414-07:002011-03-19T12:32:36.414-07:00Scott:
While the result may seem counterintuitive...Scott:<br /><br />While the result may seem counterintuitive, two things have to be considered.<br /><br />1. The lower reliability of EV goal ratio, relative to overall goal ratio.<br /><br />2. The fact that even strength ability and special teams ability are correlated skills at the team level.<br /><br />In fact, in relation to the first factor, this limitation can be accounted for by determining the true correlation between even strength outshooting (as measured by EV Tied shot ratio) and even strength outscoring (as measured by EV goal ratio). <br /><br />Using the first five post-lockout seasons as our sample, that correlation would be approximately 0.87. By comparison, the true correlation between EV Tied shot ratio and overall goal ratio is approximately 0.80 (as calculated in <a href="http://objectivenhl.blogspot.com/2011/01/even-strength-outshooting-and-team.html" rel="nofollow">this</a> post).<br /><br />So it appears that even strength outshooting is more closely tied to even strength outscoring than overall outscoring, once the imperfect reliability of the involved variables is accounted for.<br /> JLikenshttp://www.blogger.com/profile/02570453428274983835noreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-48468277726722970152011-03-19T11:58:29.913-07:002011-03-19T11:58:29.913-07:00Corsi Tied is only marginally more predictive of f...<i>Corsi Tied is only marginally more predictive of future success than goal ratio or winning percentage when looking at samples of 60 games or more. In other words, as the sample size becomes increasingly large, there are diminishing returns with respect to the predictive advantage of Corsi. By the end of the season, all three variables seem to predict future success equally well.</i><br /><br />Unless I'm misunderstanding something, isn't the more likely explanation that the future W% becomes more dispersed relative to talent when the sample decreases in size? The correlation for Corsi falls as you increase the sample for Corsi but decrease the sample for future winning percentage. <br /><br />So instead of larger samples of W% and EV GD being better predictors (their correlation coefficients decrease as the sample gets bigger) it's that future results become more random with smaller samples?Michaelnoreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-58267678251933198182011-03-19T11:01:26.701-07:002011-03-19T11:01:26.701-07:00Thanks for posting the addendum. It's very sur...Thanks for posting the addendum. It's very surprising to me that the EV Corsi results have a stronger correlation to overall performance than they do to strictly EV performance. Did not expect that at all.Scott Reynoldshttp://www.blogger.com/profile/05735545121522530577noreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-62311645763583533012011-03-18T16:15:19.948-07:002011-03-18T16:15:19.948-07:00Thanks Scott.
"If I understand you correctly...Thanks Scott.<br /><br /><i>"If I understand you correctly, the Corsi Tied metric is including only EV results, whereas the goal differential metric is including all goals, both at EV and on ST."</i><br /><br />That's correct.<br /><br /><i>"It would be interesting to see whether using EV GD would have any impact on the results."</i><br /><br />I actually ran those numbers as well. In particular, I looked at which of EV goal ratio and EV Corsi Tied was better able to predict future even strength outscoring. <br /><br />I'll post an addendum.<br /><br /><i>"The other thing that stands out is that these correlations aren't as strong as I had expected (although where that expectation came from, I'm not quite sure)."</i><br /><br />Yeah, predicting future performance is difficult.<br /><br />Early in the season, the sample over which future performance can be assessed is large. However, differentiating between the good and bad teams is hard.<br /><br />By the end of the year, the reverse is true - we're much better able to distinguish the good teams from the bad ones, but predicting future performance is still difficult because there's just not a lot of hockey to be played at that point.<br /> JLikenshttp://www.blogger.com/profile/02570453428274983835noreply@blogger.comtag:blogger.com,1999:blog-3299311926633621468.post-33554611749278658302011-03-18T11:45:20.967-07:002011-03-18T11:45:20.967-07:00Good post! If I understand you correctly, the Cors...Good post! If I understand you correctly, the Corsi Tied metric is including only EV results, whereas the goal differential metric is including all goals, both at EV and on ST. It would be interesting to see whether using EV GD would have any impact on the results.<br /><br />The other thing that stands out is that these correlations aren't as strong as I had expected (although where that expectation came from, I'm not quite sure).Scott Reynoldshttp://www.blogger.com/profile/05735545121522530577noreply@blogger.com