From Boom to Bust: Avoiding the Wrong Quarterback

In the last installment of From Boom to Bust, I compiled pre-draft scouting reports of the most successful quarterbacks since 1998 to see what traits they might have had in common coming out of college, and therefore, which ones matter the most when scouting draft prospects. That project got such an overwhelming response I decided to dig a little deeper. This time, I've taken a look at the biggest quarterback busts since 1998 to see which kinds of traits they had in common to help us predict which quarterbacks in the upcoming draft are likely to bust. When compiling the list of quarterback busts, I had to first arrive at a definition of a bust. For a quarterback to be labeled as a "bust," there has to have been an expectation that the prospect would succeed in some capacity in the NFL. So, naturally I was looking at quarterbacks drafted in the 1^st round, as their expectations are much higher than the later rounds. If a quarterback drafted in rounds 2-7 didn't pan out, it's not that big of a deal, since the expectation for them is not really that great. In fact, according to the now defunct Draftmetrics website, from 1992-2012, any player generally chosen within the first forty picks had at least a 60% chance of being a 3-year starter (48 starts), and players generally chosen between picks 1-13 additionally have a 44% chance of becoming a Pro-Bowl caliber player. Quarterbacks specifically chosen between picks 1-13 have a 68% chance to be a 5-year starter (80 starts), and if they are chosen between picks 14-40, they have a 33% chance to become a 5-year starter. All of this is to say that generally speaking, there is a very good expectation that any quarterback drafted in the 1^st round should be at least a 3-year starter in the NFL, and if they are drafted within the first 13 picks, the expectation is even greater (a 5-year starter).

So using a 3-year starter benchmark (or 5 years for a 1-13 overall pick) as the expectation to determine if a player was a bust or not, it became apparent that some players were able to accumulate a lot of starts but still weren't very good (as in, lost a lot of games, never made it to the playoffs, or the Pro Bowl and earned very little if any accolades). So in addition to tracking the number of starts, I used the Adjusted Net Yards Per Attempt metric again, because it is the efficiency metric that most closely correlates to wins (aka, success) for quarterbacks. In most years the average ANY/A is 5.35, so to be classified as a "bad" quarterback they would have needed to be below that average mark for their career. In other words, if a quarterback was a 3 or 5-year starter with an average ANY/A higher than 5.35 and drafted in the 1^st round, that would be an acceptable expectation for that pick and would therefore not get the "bust" label. As an example, Jason Campbell is someone who just barely avoids the bust label, because he meets the starts requirement of his draft position, and his ANY/A is just slightly above average. Campbell was drafted 25^th overall in the 2005 draft, so his expectation is at least 48 starts. He's started 79 games in his career, and his career ANY/A rating is 5.49, just slightly above average. So he doesn't qualify as a bust (although it's very close). If he had been drafted higher in the first round, between picks 1-13 overall, then he would have qualified as a bust.

So, if a player starts fewer years than expected based on their draft position, or had a career ANY/A lower than 5.35, then they would qualify for the bust label. Lastly, while I looked at the years 1998-2013 for the successful quarterbacks, because the cut-off of starts for the bust quarterbacks is 3 years, this data set will only look at draft years from 1998-2011, because naturally anyone drafted in 2012 or 2013 hasn't had the full 3 years to qualify as a bust or not. Also, anyone drafted in the top 13 picks of 2010 or 2011 will not be considered, since they haven't had a full five years in the league to qualify either.

Armed with these criteria, I arrived at a list of 17 quarterbacks drafted in the first round between the years 1998 and 2011 that would qualify as a bust. The table below shows all 17 busts, sorted by the year in which they were drafted, including what selection overall. This is not a pretty list.

Notice, there is not a single quarterback who won the Super Bowl as a starter (although Byron Leftwich was the backup QB for the Steelers during one of their Super Bowl wins). Speaking of Byron Leftwich he barely makes the cut as a quarterback bust. His career ANY/A rating of 5.51 is just slightly above average and he has a couple of playoff games under his belt, however as the 7^th overall pick, he was expected to start at least 84 games and he only made it to 50 with no other accolades to speak of. Some may argue his ankle injuries were to blame, but Leftwich couldn't beat out Joey Harrington (another bust) while in Detroit, which speaks to his level of bust-worthiness. While some may disagree with a few of the selections above, I'm confident that they are all busts due to the expectations associated with where they were drafted in the 1^st round. Also, players like Blaine Gabbert and Christian Ponder seem destined to be added to this list in another year or two.

In any case, like I did with the successful quarterbacks, I scoured the internet for Pre-draft scouting reports of these 17 players and I was able to find something on all of them. I tracked all the attributes mentioned in the reports (both good and bad) to see if there were any commonalities, particularly with the negative attributes. Since so many people asked in the previous article, I thought I'd mention that I used scouting reports from the following sources: Sports Illustrated, Fox Sports, Scout.com, CBS Sports as well as a few others. Just like with the set of successful quarterbacks, there wasn't much available and in most cases I was only able to find one report for each quarterback. In any case, after compiling all the reports, I found that there were several mildly common negatives traits these busts had in common. They were:

1. Forces passes into tight coverage and/or stares down receivers (7 out of 17)
2. Lacks accuracy (6 out of 17)
3. Poor mechanics or fundamentals (6 out of 17)

Now, it's worth pointing out that none of these traits are common for the majority of the quarterback busts, but 41% of the busts exhibited forcing passes into tight coverage. For the successful quarterbacks, the most common negative trait: lacks mobility, applied to 40% (making it something of a false negative), but none of the other negative traits applied to more than 20% of the successful QBs. So for negative traits #2 and #3 above to also apply to 35% of the quarterback busts makes me think that they do matter a little bit, especially considering that accuracy was one of the common positive traits of successful quarterbacks. That leads me to believe that the single most important trait for whether or not a quarterback will succeed or bust, is their accuracy since it appears on both metrics and will swing a prospects point total dramatically into the positive or negative more than any other trait.

In looking at the positive attributes of the quarterback busts, I did find some very interesting common traits. In thinking about what that means for these busts to actually find common ground in something viewed as a positive, it can only mean one thing: these traits are false positives. In other words, these positive traits are not important to predicting success for a quarterback prospect since they were common for so many quarterback busts. There were only 2 positive traits of quarterback busts that overlapped with the successful quarterbacks: leadership/competitiveness and having a strong arm. In any case, here is the list of false positives:

1. Good Size and strength (11 out of 17)
2. Strong competitor and leader (11 out of 17)
3. Strong arm (8 out of 17)
4. Quick delivery/release (8 out of 17)

The list of common positives isn't nearly as long as for the bust quarterbacks as it was for the successful ones, and there was much less commonality as well, but that doesn't mean they aren't significant. The most common positive for successful quarterbacks (pocket awareness) was common among 75% of the successful quarterbacks; whereas the most common false positive among bust quarterbacks (good size/strength) was common for 65% of the busts. There were several other false positives that are worth mentioning, but don't quite make the cut: sturdy/durable and good athleticism. In trying to make sense of what these false positives mean I've come to the conclusion that many of the quarterbacks that turned out to be busts were drafted so highly because of their false positives: like prototypical size and arm strength (JaMarcus Russell, J.P. Losman, Byron Leftwich, etc). However, because they were lacking the most important positives like accuracy or decision-making, or had common negatives of busts (like staring down receivers) they weren't able to succeed. In other words, simply looking the part and being a good athlete is not necessarily a recipe for success in the NFL. A quarterback has to be able to read an NFL defense and throw an accurate ball without staring down their intended target too.

In the last installment I developed a metric to give a quarterback prospect a success rating, or in other words a way to rank the prospects by who is most likely to succeed based on how many traits they have in common with other successful NFL quarterbacks. So, below is the opposite metric for a bust rating. This can be used to rank the prospects based on which ones are most likely to bust, based on their similarities with other quarterbacks who have busted. This one is based around the negatives first, followed by the false positives. Since there are 7 traits total, the most common trait receives 7 points and so on down the line, scored by frequency of appearance in the scouting reports.

Bust Traits Metric

Negatives
#1. Forces passes into tight coverage and/or stares down receivers (-7 points)
#2. Lacks accuracy (-6 points)
#3. Poor mechanics or fundamentals (-6 points)
False Positives
#4. Good size and strength (-5 points)
#5. Good leader and competitor (-5 points)
#6. Strong arm (-4 points)
#7. Quick delivery/release (-4 points)
Total: -37 points

So do these 7 traits actually predict potential NFL busts? Well that remains to be seen. Ben Goessling over at ESPN put together this really nice article a few weeks ago about drafting quarterbacks in the NFL, and his conclusion is that drafting a quarterback essentially comes down to getting lucky. While this may be the case, it doesn't mean we shouldn't still do our homework and try to increase our chances to get lucky. Any general manager can simply call out a name on draft day, but the truly great ones have done their research and try to minimize risk as much as possible. And I find it pretty compelling that these quarterback busts did have some common traits coming out of college that should have been red flags for any organization, either in the way of negative traits that have a bigger impact on their abilities, or an over-valuing of the "false positives."

Another interesting aspect about this new bust data is that false positives #5 and #6 are traits that successful quarterbacks also exhibited. In the Success Metric, being a good leader and competitor earned prospects 5 points, but in the Bust Metric it will earn them -5 points. In other words, being a good leader will have exactly zero impact on a quarterback's ability to be successful or bust in the NFL, because it was common among both successful quarterbacks and busts. In other words, it's just not an important trait to distinguish one type of quarterback from the other. Similarly, having a strong arm will have less impact too. In the Success Metric a strong arm earned prospects 5 points, but in the Bust Metric it earned them -4 points for an overall total of 1 point. In other words, having a strong arm will have some minimal impact towards being successful, but not nearly as much as the other traits, like having good pocket awareness, reading defenses and making good decisions. And this just seems to make a lot of common sense too.

There was a lot of excellent discussion in the comments section of the first installment of From Boom to Bust that led me to completely reassess the scouting reports for both the original success metric and this new bust metric. First off, general accuracy in scouting reports was a major concern, so I have decided to look at more than one report in an attempt to cull more variables together and remove some inherent biases. Second, I wasn't consistent in the way I looked at the scouting reports, and this time I have decided to use the same two sources for all prospects: one mainstream source (CBS Sports) and one lesser known source (The Sideline View).

The new bust metric provides a somewhat compelling case for its usage when you run it on both the successful quarterback's data set, and the bust quarterbacks. Because of the small overlap in the two positive traits mentioned above, the successful QBs naturally scored more bust points than seems logical. However, in looking at the average bust scores we begin to see a difference. First off, the successful QBs did not generally receive any negative marks for accuracy, although a few had "deep ball accuracy" mentioned as a negative. Also, with the exception of 1 quarterback, no successful QB had their fundamentals or mechanics called into question. In other words, while there were common negative traits of bust QBs, the successful QBs did not share those same traits. On average, the successful QBs scored -12.3 out of a possible -37 on the bust metric (mostly because of the overlap), while the bust QBs averaged a score of -17.1 out of a possible -37. So the Bust Metric does seem to have a small impact in differentiating the two data sets. So (get on with it!) if we apply the bust metric to the 2014 quarterback prospects, we get some unexpected results. Here are the results applied to the consensus Top 10 QBs from the two sources I consulted, sorted by those quarterbacks who have traits that are the most common with the 17 quarterbacks who busted in the table above.

Keep in mind this bust metric is only one half of the equation. It's alarming that Blake Bortles and Johnny Manziel, who are both graded as 1^st round draft picks, score so highly in the bust metric. The others, like Garoppolo and Mettenberger aren't as big of a deal, since they don't have 1^st round grades, but it would still be wise to avoid both of them too.

Based on the scouting reports of Blake Bortles, I have seen a lot made of his size and arm strength, and how he "looks the part", but there was very little discussion of what a great quarterback he actually is, right now. Most say his "potential" is good, and he has "upside", but he has a lot of the same types of traits that the 17 bust quarterbacks possessed. In the case of Johnny Manziel, often in the conversation as a 1^st round prospect, he also exhibit many bust-able quarterback traits, like forcing passes into coverage and relying on his receivers to bail him out, and the false positives like "strong arm", and "quick release". Derek Carr falls somewhere in the middle. His score of -16 is better than the average bust quarterback, but still noticeably worse than the average successful quarterback. In other words, it's acceptable, but not great. He only had one consistent false positive trait (strong arm), but his negatives were a little alarming: forces passes and lacks accuracy. Still, he's better off than Bortles in the bust metric overall. Notably, Teddy Bridgewater (the only other quarterback in this draft class with a 1^st round grade) has an insanely low bust rating only scoring points on the "Good Leader" false positive, with no other bust worthy traits. So therefore, Bridgewater would be the safest quarterback pick in the draft with virtually no bust potential.

So, what about that other half of the equation, the success metric? In looking back at the Success Metric from the previous article, all 7 of those success traits showed up in the scouting reports for the 17 bust quarterbacks, although most of them did not possess a majority of the seven traits. In fact, running the "success" metric through both sets of past quarterbacks revealed an even bigger difference than the bust metric. The 20 successful ones averaged a score of 22.5 out of a possible 35 on the success metric, while the 17 bust quarterbacks averaged a score of only 12.2, almost half as high. This speaks to the value of the success metric in sorting out quarterback destinies and it would seem to be an even better indicator of potential success in the NFL. Again, the successful quarterbacks averaged -12.3 on the bust metric, while the bust quarterbacks averaged -17.1. So, if we combine the two metrics together, this means if a draft prospect can score an overall total of 10 or more points they will be on par with the average of successful QBs. On the flipside, if they score -5 or worse points when combining both metrics, they will be on par with the average of bust QBs. This gives us a pretty good baseline to compare the 2014 prospects with the data set of successful QBs and bust QBs from the past.

To recap, here are the two metrics in case you wanted to run some numbers on your own:

So, consider the following ranking to be the final "Boom or Bust" ranking of the quarterbacks that takes into account this new bust data, and a redo of the success metric.

In what should come as no surprise, Teddy Bridgewater tops the list easily, with the lowest bust score by a mile and the 2^nd highest success score. His overall total is way above the average of successful NFL quarterbacks, and despite having a bizarrely bad Pro Day recently, he should be regarded as the top quarterback prospect in this draft. And it's not particularly close. As a prospect his overall total score is right on par with prospects like Russell Wilson, Philip Rivers, Robert Griffen III, and Andrew Luck, and this metric paints him out to be better than Aaron Rodgers, Peyton Manning, and Drew Brees when they were coming out of college. But notice that the drop-off in this draft class from Bridgewater is huge. Only three other quarterbacks even project in the positive: Derek Carr, AJ McCarron and Blake Bortles. As was mentioned above, Bortles has the highest bust rating of any quarterback, but with the revised success metric, he also has the highest success rating. There is a reason so many scouts, NFL GMs and draft ranking sources view him as the #1 quarterback prospect: he shares a lot of the same elite traits as the most successful quarterbacks in the NFL. But, so did Brady Quinn and JaMarcus Russell and yet they turned out to be busts. In the case of Quinn and Russell, who both scored highly in the success metric (35 and 20 respectively), it was their bust metric that did them in (Quinn scored a -18 and Russell had a -21). So, I would exercise extreme caution when it comes to Blake Bortles...after all, most scouts have mentioned that he isn't ready to start right away in the NFL anyway. His potential gives him a high success score, but the fact that he hasn't been able to put it all together yet, raises massive red flags.

AJ McCarron is one the least talked about prospects this year, and yet he is ranked 3^rd in this metric. His success score is respectable, and his bust metric is reasonable, and yet his overall score puts him somewhere in-between a successful QB and a bust. Think of him as the next Jason Campbell or Kerry Collins: a guy that can find some level of success, but will probably never be elite. That said, as a 2^nd or 3^rd round prospect, he would represent some excellent value, and as I mentioned in the first article he possesses many of the traits that Norv Turner would want in an Air Coryell system. I would guess the Vikings are interested in McCarron (and this is pure speculation on my part) because they did have a huge scouting contingent at Alabama's Pro Day back in March.

This metric was not kind to Jimmy Garoppolo or Zach Mettenberger, who round up the bottom of the list. So, if they were chosen in the 1^st round they will almost surely fail to meet their expectations as a 1^st round pick and turn out to be a bust. However, both are projected as 2^nd round or later picks, so I'm not as concerned. Still, if we chose either one with the #40 selection, technically they would meet draft metrics expectation of being a 3-year starter, and I don't think either one will succeed in that regard. Both have topped the scales of the bust metric, and are lacking pretty severely in the success metric, so it would be wise to stay away.

And then there is Johnny Manziel who ends up in the negative overall. As mentioned above his bust potential is pretty high, and unfortunately he doesn't possess enough traits on the success metric to make up for it. Perhaps someone can find some better scouting reports that would improve his numbers, but the two I looked at aren't favorable. In fact, the two scouting reports agree on only two things for Manziel: he can throw well on the run, and he forces passes into coverage. Like I mentioned in the last article, there is a lot of disagreement on Manziel so he might be a candidate to explore further.

Lastly, many readers commented about wanting to see these metrics applied to previous draft classes, and that is the next major step with this project. I haven't found the time to run the numbers on all of the previous draft classes, but I did want to share how Christian Ponder would have fared in these updated metrics. In the last article I used his scouting report from Walterfootball.com and received plenty of criticism for that decision. He scored poorly in the success metrics with that scouting report. So, this time I've decided to run his scouting reports from NFL.com and Draft Insider through the metrics (mainly because that's about all I could find). After averaging those two reports together, he scored a 6 on his success metric and a -10.5 on his bust metric for an overall score of -4.5, squarely in the bust range overall. His bust metric in-and-of itself wasn't that bad actually. In fact, he would have had the 3^rd lowest bust score of all of the 2014 prospects and was a pretty low risk pick in that regard, but he severely lacked the traits necessary to succeed as an NFL quarterback as he would have had the 2^nd lowest success score of the ten 2014 prospects. As I mentioned before, he was not a wise 1^st round selection in the 2011 draft.

So, where does that leave us? Well, after working through more of this data I'm convinced that this draft class consists of Teddy Bridgewater, and then a whole lot of "meh". Because Bridgewater scores so highly in the metric, I would advise the Vikings to strongly consider moving up in the draft for Bridgewater as he represents the most value of any prospect and his scores in the metric imply that a trade up for him would be worth it. If we don't trade up, he will almost certainly be taken by the time we are on the clock with the 8^th pick, and in that case then I would be perfectly fine with drafting the 2^nd best prospect, Derek Carr (assuming he is also still available). I would avoid Blake Bortles and Johnny Manziel like the plague, even if they fall to #8 due to their alarmingly high bust score. If we pass on a QB in the 1^st round, then taking AJ McCarron in the 2^nd round would be the next best option as I would avoid Mettenberger and Garoppolo outside of a mid-round or later selection. I do think the Vikings need to take a quarterback in the 1^st or 2^nd round, and the million dollar question is: who will be available, and when are these prospects going to get taken? For the answer to that we'll just have to wait until May 8^th.