From Boom to Bust: Testing the Quarterback Metric

This past off-season I have been scouring current and past scouting reports to try to develop a metric that we can use to evaluate quarterback prospects.  I started by developing a metric to evaluate the traits of successful quarterbacks.  I cataloged the traits found in pre-draft scouting reports of an elite list of 25 successful quarterbacks that have been drafted since 1998, and based the metric on those traits that were most common among that pool of players.  In other words, I attempted to answer the question, "What common traits did the most successful quarterbacks in the NFL have coming out of college?" Then I went back and re-evaluated the "success metric" based on excellent feedback from the readers here at the Daily Norseman.  I also developed a second metric to evaluate the traits of quarterback busts.  It was the same process, except that I catalogued the common traits of the 17 quarterback busts since 1998 and based the bust metric on those traits that were most common among those players.   That led me to the final Boom or Bust metric, which you can also find in that second link (and is listed below).  The last step in this process is what you'll find here: verifying the accuracy of the metric.  I have gone back and run the metric on quarterbacks drafted in the 1^st round of past drafts to see how successful it would have been at predicting the future successes of those players.  The short of it is: it's more accurate than a random guess.  It's not fool-proof mind you, but over the course of seven drafts from 2004 through 2010, it would have accurately predicted which 1^st round quarterbacks would bust and which would be serviceable or better 73% of the time.  Why did I only go back to 2004?  Well, I really wanted to use at least two scouting reports for every quarterback when testing the metric to ensure better accuracy, but the farther back in time I went, the harder and harder it was to find reliable scouting reports online.  I wasn't able to track down more than one reliable scouting report for the quarterbacks drafted in 2003 and earlier, so there really is no other reason than that.  I stopped at 2010, because a quarterback needs at least 4 years in the league to qualify as a bust or not, and those quarterbacks drafted in 2011 and later haven't had a full 4 years yet.

So, during the time frame of 2004 - 2010 there were 19 quarterbacks drafted in the first rounds.  I went back and found two pre-draft scouting reports for all nineteen of these quarterbacks, which included reports from the NFL Draft Scout (reprinted at NFL.com and CBS Sports), Draft Breakdown, Sports Illustrated, Pro Football Weekly, Football's Future, About.com, Walter Football and a few others.  I ran the metric through both scouting reports for each player and averaged the results together from both reports.  As a reminder, here is the Boom or Bust metric for evaluating quarterbacks:

So if one of these traits is mentioned in the scouting report, that quarterback receives the requisite point totals.  In looking back at the average scores of successful and bust quarterbacks, the following point range can give you a good idea of what kind of expectation and career prediction to have for a quarterback based on their point totals in each metric.

It's worth pointing out that in this particular data set (2004-2010), the Bust Metric by itself was almost as accurate overall as the combined metric in predicting the future of these quarterbacks and was 68% accurate by itself (although they each had slightly different results on a per quarterback basis).   The success metric by itself was a little less accurate, correctly predicting the future only 61% of the time.  In any case listed below are the 19 first round quarterbacks drafted between 2004 and 2010, with their metric scores from their pre-draft scouting reports and pre-draft prediction.  I have taken some leeway in assigning the outcome score to this.  My biggest concern in all of this is to ensure that if the metric predicts the quarterback to be in the bust category that they truly are a bust.  After that, we can end up splitting hairs all day about what makes a quarterback "average" or "successful" or not.  In other words, if the metric predicts that a quarterback will be merely league average, but he turns out to be a successful one then I'll still call it a win for the metric, because it didn't predict that quarterback to bust.  I think teams are mostly concerned with not having their 1^st round quarterback bust (like JaMarcus Russell or Ryan Leaf), than whether or not they get a Jason Campbell versus Aaron Rodgers type.  I have given each quarterback an outcome label of "yes", "maybe" or "no".  A "maybe" label essentially means that the player has performed reasonably well, but still has enough time left in their career to qualify for their prediction label.  In those cases, the quarterback receives half-credit for their outcome.

As you can see, Sam Bradford and Alex Smith score a "maybe" in the accuracy column.  The Boom or Bust metric predicts both of them to land in the "successful" category, that is, ending their careers with a Top 32 ANY/A rating, appearing in at least 10 playoff games or winning the Super Bowl (what I used to label a quarterback as "successful" in the first place).  Neither quarterback has achieved that yet, but both still could reach those lofty marks by the time their careers are over.  I wouldn't call either quarterback a bust yet, so they aren't quite a yes or a no in the accuracy label.  It's also worth mentioning that the metric predicted Jay Cutler and Joe Flacco to be merely average, but both of them qualify as being successful.  I still call this "yes" in outcome though, since neither quarterback busted.  Unfortunately there were a few misses with the metric, although nothing that is way off the mark.  The metric missed on Brady Quinn, Matt Leinart, and Mark Sanchez.  The scouts liked all three, predicting them all to have decently average careers as starting quarterbacks in the NFL, but unfortunately they all busted.  At least Quinn's expectations were a bit lower than Leinart and Sanchez as he was drafted near the bottom of the 1^st round in 2007. And to the metric's credit, it didn't predict any of them to be elite, successful quarterbacks (but that didn't stop teams from drafting Leinart and Sanchez in the top 13 which was a mistake based on their prediction).  On the flipside, Jason Campbell was predicted to be a bust, and while that might be debatable he's had the quintessential league average career thus far, especially as a quarterback drafted 25^th overall.

So, this metric isn't fool-proof by any means, but being 73% accurate is pretty significant.  So, I think it's good to take another look at the combined metric scores of the 2014 quarterback class.  I'm using the same list I used in the previous article, but this time I have included the prediction label based on their combined metric scores and their bust metric scores.

There are two things to pay attention to here.  The first is their combined score and the prediction label.  Based on this metric there is only one quarterback in this draft class that this metric predicts to be an elite, successful quarterback and is therefore worthy of a Top 13 pick: Teddy Bridgewater.  And should Mettenberger or Garappolo be drafted in the first round, this metric predicts them to bust.  Everyone else is destined for a league average career in some capacity and could be worthy of a 1^st round pick, but not in the Top 13.  That said, obviously not all of these quarterbacks will be drafted in the 1^st round so "average" will be different depending on where they are drafted and it's worth pointing out that there is a huge difference in range between Derek Carr who scores a positive 4.5 and David Fales who scores a -4.  Both are on the extreme opposite edges of the "average" spectrum.   Secondly, I included another prediction column at the end that is based solely on the bust metric.  Because the bust metric was nearly as accurate in predicting the future as the combined metric (albeit with different results), you can see that Blake Bortles and Johnny Manziel also have high potential to bust in this draft class.  It's worth pointing out that neither is predicted to have a particularly successful career in the NFL with this metric in that the combined metric predicts them to be only average, while the bust metric predicts them to bust.  In either case, it's not a ringing endorsement for them to be a Top 13 pick in the NFL and I would even be wary of spending a 1^st round pick on either one.  Another interesting name to consider on this list is Brett Smith out of Wyoming.  The combined metric has him as a league average prospect, but his bust metric is appreciably low which gives him very low risk.  I would not spend a first round pick on him though as his success score is pretty low, but as a mid-round sleeper the Vikings could do a lot worse than Brett Smith.

All that said, with only a 73% accuracy rating and 10 names on the list, odds are the metric will miss on about 2-3 of these prospects.  Your guess on which ones it will miss is as good as mine.  If I had to choose quarterbacks that I would definitely want the Vikings to avoid though, it would be Bortles, Manziel, Mettenberger and Garoppolo as their bust potential is just much too high for my tastes.  After Teddy Bridgewater, I'm not particularly fond of any of the quarterbacks in the 2014 draft, and if this metric is to be believed, this looks to be a pretty weak class at the top end, but a decently deep draft in the middle.  If the numbers are to be believed, this draft should produce a fair number of good backup quarterback options and a few league average options, but not many elite, successful quarterbacks.