Where Chris Boyle's Shot Quality work can and probably can't be useful.

Chris Boyle has been slowly unveiling the results of his shot quality tracking for Sportsnet over the past few weeks.  Like a lot of smart people, Boyle has recognized an area where the NHL is extremely poor at tracking something in today's NHL hockey games - in this case, where shots on goal are taken from on the ice - and attempted to fix the situation and watch games himself and track where each shot comes from.  Boyle had previously shown that the NHL shot trackers are hopelessly incompetent and this is a pretty nice way to try and get better data.

So Boyle has tracked a ton of hockey games and marked down where shots have come from - and, perhaps even more interestingly, what types of shots these shots were (Clean shots, Shots off of passes, shots off of deflections, and shots off of rebounds).  This is all very cool stuff and can be useful if we have a large enough sample and we apply it in the proper places.

Unfortunately, so far Boyle hasn't quite done that.  After finding the average SV%s of each type of shot, Boyle posted today his results as to several goaltenders, focusing mainly upon James Reimer.  Boyle finds that Reimer's first season's results may have been skewed in Reimer's favor by the sheer # of clean shots Reimer faced.  In other words, Reimer faced low quality shots more often than average, resulting in an inflated SV% that we might expect to drop as Reimer faces higher quality shots in the future.  So far, this is pretty interesting stuff, especially when we're dealing with small goalie sample sizes as a single season.

Reimer's season in 2012-2013 was different as per Boyle's findings - Reimer featured a much tougher shot selection, but still had way above average results, particularly against rebound shots.  The money graphic is this one:

According to Boyle, League average rebound SV% (at 5 on 5) is .760.  After two years of being below average at this mark, Reimer actually hit a rebound SV% of .844 in the last 1196 shots.  And here is where Boyle goes way off the deep end:

His clean save percentage remains fairly static, but his rebound and transition rates continue to improve.

All of this could be small-sample magic, but it is consistent in regards to most young goaltenders’ learning curves. The biggest adjustment a goaltender makes during the adjustment period to the NHL is the speed of the game. When overwhelmed by the pace, he is late tracking the puck and will trail the play. When that happens, he must rely on reflex and reaction. That leads him to play with less control and means more of a struggle to anticipate and read the play.

As a young goalie gains experience, he relies less on reflex and reaction saves because he tracks the puck better. This allows him to skate ahead of the play and employ a game plan. Reimer has quieted his game and with that his deflection and rebound save percentages have risen thanks to better positioning.

The Reimer of 2010-11 could not have survived the onslaught of rebound opportunities he faced in ’12-13—he saw twice the rebound opportunities, yet maintained the exact same goal rate.

.....

With a strong start to 2013-14, Reimer’s numbers have climbed to a point where he is not just a reliable NHL starter, but a good one. Even if he remains weak in transition, maintaining a dominant performance when faced with rebound shots will push him above the middle class.

 Now Boyle - to his credit - gets it right in the final few paragraphs of the piece, but here he's gone incredibly wrong: He's assuming that goalies improve in a certain way (apparently due to his own past experience) and that's what is the cause here, rather than noting how improbable this improvement actually is.  Quoting the correct passages later: 

" Only five of the 30 goaltenders studied have been able to register a +.800 on rebounds and two of them (Price and Lundqvist) couldn’t maintain it the following season"

 In other words, Reimer's save % is based upon an incredibly fluky rebound save % over a small sample that is unlikely to continue!  That's not a sign that Reimer has improved and can actually  handle rebounds now which he couldn't back then!  By the way, the converse here is also true: it's certainly possible Reimer's poor rebound sv% back in 2010-2011 was ALSO fluky in the reverse direction! 

None of this is to say that Reimer may not be an above average goaltender.  But none of the data presented by Boyle makes that clear to us any more than the classic save percentage measure.  This is because the data doesn't solve the issue of showing us whether something is REPEATABLE or something that is just an instance of unlikely play to continue (as is likely the case with Reimer).

BUT:  BOYLE'S WORK CAN BE INCREDIBLY USEFUL

I hope Boyle will go into this in the future, because as I hint at above, this isn't to say his work doesn't add something.  For one, knowing that Toronto faced harder shot quality last year than 2 years prior is certainly fascinating given the Leafs vs Analytics war that's been going on.

But the information is still useful as a goaltender evaluation tool (even beyond simply helping team coaches work with goalies on weak areas).  Again, Boyle's data suggests that Reimer's 2010-2011 season #s are inflated and were not likely to repeat, and that's extremely big information that could help us smoke out goalies who are the next Steve Mason as opposed to the next Henrik Lundqvist. 

Essentially, this is the equivalent to baseball stats like batted ball stats (LD%, GB%, FB%) or BABIP, which - if they were reliable (Batted Ball Stats sadly are less reliable than one would like) - would help us know when certain players are getting lucky or they're actually managing to show something that is repeatable skill. 

Here it's unlikely that Boyle's #s can really show something as repeatable skill more so than our existing measures.  But they can show when certain results are fluky and bear more watching more than we could before - instead of simply saying that goalies after X amount of shots regress Y amount, we can see whether a goalie's #s are particularly likely to regress due to shot quality against regressing.  That's what's genuinely interesting here.  And that's why I hope Boyle releases the raw data and am still interested in the project.