From Boom to Bust: Testing the Quarterback Metric

Jamie Rhodes-USA TODAY Sports

In the third installment of From Boom to Bust, the Daily Norseman runs the quarterback metric through the ringer to test it's accuracy in predicting successful and bust quarterbacks. As it turns out, this metric is better than throwing a dart at a list of names.

This past off-season I have been scouring current and past scouting reports to try to develop a metric that we can use to evaluate quarterback prospects.  I started by developing a metric to evaluate the traits of successful quarterbacks.  I cataloged the traits found in pre-draft scouting reports of an elite list of 25 successful quarterbacks that have been drafted since 1998, and based the metric on those traits that were most common among that pool of players.  In other words, I attempted to answer the question, "What common traits did the most successful quarterbacks in the NFL have coming out of college?" Then I went back and re-evaluated the "success metric" based on excellent feedback from the readers here at the Daily Norseman.  I also developed a second metric to evaluate the traits of quarterback busts.  It was the same process, except that I catalogued the common traits of the 17 quarterback busts since 1998 and based the bust metric on those traits that were most common among those players.   That led me to the final Boom or Bust metric, which you can also find in that second link (and is listed below).  The last step in this process is what you'll find here: verifying the accuracy of the metric.  I have gone back and run the metric on quarterbacks drafted in the 1st round of past drafts to see how successful it would have been at predicting the future successes of those players.  The short of it is: it's more accurate than a random guess.  It's not fool-proof mind you, but over the course of seven drafts from 2004 through 2010, it would have accurately predicted which 1st round quarterbacks would bust and which would be serviceable or better 73% of the time.  Why did I only go back to 2004?  Well, I really wanted to use at least two scouting reports for every quarterback when testing the metric to ensure better accuracy, but the farther back in time I went, the harder and harder it was to find reliable scouting reports online.  I wasn't able to track down more than one reliable scouting report for the quarterbacks drafted in 2003 and earlier, so there really is no other reason than that.  I stopped at 2010, because a quarterback needs at least 4 years in the league to qualify as a bust or not, and those quarterbacks drafted in 2011 and later haven't had a full 4 years yet.

So, during the time frame of 2004 - 2010 there were 19 quarterbacks drafted in the first rounds.  I went back and found two pre-draft scouting reports for all nineteen of these quarterbacks, which included reports from the NFL Draft Scout (reprinted at NFL.com and CBS Sports), Draft Breakdown, Sports Illustrated, Pro Football Weekly, Football's Future, About.com, Walter Football and a few others.  I ran the metric through both scouting reports for each player and averaged the results together from both reports.  As a reminder, here is the Boom or Bust metric for evaluating quarterbacks:

Success Metric - Positive Traits

Bust Metric - Negative Traits

1. Good pocket awareness, with  ability to "step up"

7 points

1. Forces passes into coverage, or stares down receivers

-7 points

2. Smart and makes good decisions with the football

6 points

2. Lacks Accuracy

-6 points

3. Good arm strength

5 points

3. Poor mechanics or fundamentals

-6 points

4. Good leader and competitor

5 points

Bust Metric - False Positive Traits

5. Good vision, and can read NFL defenses

4 points

4. Good size and strength

-5 points

6. Good accuracy

4 points

5. Good leader and competitor

-5 points

7. Can throw on the run

4 points

6. Strong arm

-4 points

Note: Subtract point total for any traits listed as a negative

7. Quick release/delivery

-4 points

Success Metric Total Possible

35 points

Bust Metric Total Possible

-37 points

So if one of these traits is mentioned in the scouting report, that quarterback receives the requisite point totals.  In looking back at the average scores of successful and bust quarterbacks, the following point range can give you a good idea of what kind of expectation and career prediction to have for a quarterback based on their point totals in each metric.

Success Metric

Bust Metric

Combined Metrics

35 through 22.2 = Successful

0 through -12.2 = Successful

35 through 10 = Successful

22.1 through 12.3 = Average

-12.3 through -17.1 = Average

9.8 through -4.8 = Average

12.2 through 0 = Bust

-17.2 through -37 = Bust

-5 through -37 = Bust

It's worth pointing out that in this particular data set (2004-2010), the Bust Metric by itself was almost as accurate overall as the combined metric in predicting the future of these quarterbacks and was 68% accurate by itself (although they each had slightly different results on a per quarterback basis).   The success metric by itself was a little less accurate, correctly predicting the future only 61% of the time.  In any case listed below are the 19 first round quarterbacks drafted between 2004 and 2010, with their metric scores from their pre-draft scouting reports and pre-draft prediction.  I have taken some leeway in assigning the outcome score to this.  My biggest concern in all of this is to ensure that if the metric predicts the quarterback to be in the bust category that they truly are a bust.  After that, we can end up splitting hairs all day about what makes a quarterback "average" or "successful" or not.  In other words, if the metric predicts that a quarterback will be merely league average, but he turns out to be a successful one then I'll still call it a win for the metric, because it didn't predict that quarterback to bust.  I think teams are mostly concerned with not having their 1st round quarterback bust (like JaMarcus Russell or Ryan Leaf), than whether or not they get a Jason Campbell versus Aaron Rodgers type.  I have given each quarterback an outcome label of "yes", "maybe" or "no".  A "maybe" label essentially means that the player has performed reasonably well, but still has enough time left in their career to qualify for their prediction label.  In those cases, the quarterback receives half-credit for their outcome.

Quarterbacks

Year Drafted

Success Metric

Bust Metric

Combined Metric

Metric Prediction

Accurate Outcome?

Sam Bradford

2010 (1)

21.5

-7

14.5

Successful

Maybe

Tim Tebow

2010 (25)

17

-33

-16

Bust

Yes

Matthew Stafford

2009 (1)

17.5

-14.5

3

Average

Yes

Mark Sanchez

2009 (5)

22

-13.5

8.5

Average

No

Josh Freeman

2009 (17)

3.5

-20

-16.5

Bust

Yes

Matt Ryan

2008 (3)

9.5

-13.5

-4

Average

Yes

Joe Flacco

2008 (18)

15.5

-11

4.5

Average

Yes

JaMarcus Russell

2007 (1)

0.5

-17.5

-17

Bust

Yes

Brady Quinn

2007 (22)

23

-16.5

6.5

Average

No

Vince Young

2006 (3)

6.5

-17

-10.5

Bust

Yes

Matt Leinart

2006 (10)

13.5

-8

5.5

Average

No

Jay Cutler

2006 (11)

3.5

-2.5

1

Average

Yes

Alex Smith

2005 (1)

25

-5

20

Successful

Maybe

Aaron Rodgers

2005 (24)

26

-11.5

14.5

Successful

Yes

Jason Campbell

2005 (25)

7

-14

-7

Bust

No

Eli Manning

2004 (1)

30.5

-15.5

15

Successful

Yes

Philip Rivers

2004 (4)

25

-21

4

Average

Yes

Ben Roethlisberger

2004 (11)

23.5

-9

14.5

Successful

Yes

J.P. Losman

2004 (22)

5

-21

-16

Bust

Yes

As you can see, Sam Bradford and Alex Smith score a "maybe" in the accuracy column.  The Boom or Bust metric predicts both of them to land in the "successful" category, that is, ending their careers with a Top 32 ANY/A rating, appearing in at least 10 playoff games or winning the Super Bowl (what I used to label a quarterback as "successful" in the first place).  Neither quarterback has achieved that yet, but both still could reach those lofty marks by the time their careers are over.  I wouldn't call either quarterback a bust yet, so they aren't quite a yes or a no in the accuracy label.  It's also worth mentioning that the metric predicted Jay Cutler and Joe Flacco to be merely average, but both of them qualify as being successful.  I still call this "yes" in outcome though, since neither quarterback busted.  Unfortunately there were a few misses with the metric, although nothing that is way off the mark.  The metric missed on Brady Quinn, Matt Leinart, and Mark Sanchez.  The scouts liked all three, predicting them all to have decently average careers as starting quarterbacks in the NFL, but unfortunately they all busted.  At least Quinn's expectations were a bit lower than Leinart and Sanchez as he was drafted near the bottom of the 1st round in 2007. And to the metric's credit, it didn't predict any of them to be elite, successful quarterbacks (but that didn't stop teams from drafting Leinart and Sanchez in the top 13 which was a mistake based on their prediction).  On the flipside, Jason Campbell was predicted to be a bust, and while that might be debatable he's had the quintessential league average career thus far, especially as a quarterback drafted 25th overall.

So, this metric isn't fool-proof by any means, but being 73% accurate is pretty significant.  So, I think it's good to take another look at the combined metric scores of the 2014 quarterback class.  I'm using the same list I used in the previous article, but this time I have included the prediction label based on their combined metric scores and their bust metric scores.

Quarterback

School

Bust Score

Success Score

Combined Score

Combined Prediction

Bust Metric Prediction

Teddy Bridgewater

Louisville

-5

23.5

18.5

Successful

Successful

Derek Carr

Fresno State

-16

20.5

4.5

Average

Average

AJ McCarron

Alabama

-13

15.5

2.5

Average

Average

Blake Bortles

Central Florida

-23.5

25.5

2

Average

Bust

Johnny Manziel

Texas A&M

-19

17

-2

Average

Bust

Aaron Murray

Georgia

-12.5

10.5

-2

Average

Average

David Fales

San Jose State

-13

9

-4

Average

Average

Brett Smith

Wyoming

-8.5

4.5

-4

Average

Successful

Zach Mettenberger

LSU

-21.5

12.5

-9

Bust

Bust

Jimmy Garoppolo

Eastern Illinois

-22.5

12.5

-10

Bust

Bust

There are two things to pay attention to here.  The first is their combined score and the prediction label.  Based on this metric there is only one quarterback in this draft class that this metric predicts to be an elite, successful quarterback and is therefore worthy of a Top 13 pick: Teddy Bridgewater.  And should Mettenberger or Garappolo be drafted in the first round, this metric predicts them to bust.  Everyone else is destined for a league average career in some capacity and could be worthy of a 1st round pick, but not in the Top 13.  That said, obviously not all of these quarterbacks will be drafted in the 1st round so "average" will be different depending on where they are drafted and it's worth pointing out that there is a huge difference in range between Derek Carr who scores a positive 4.5 and David Fales who scores a -4.  Both are on the extreme opposite edges of the "average" spectrum.   Secondly, I included another prediction column at the end that is based solely on the bust metric.  Because the bust metric was nearly as accurate in predicting the future as the combined metric (albeit with different results), you can see that Blake Bortles and Johnny Manziel also have high potential to bust in this draft class.  It's worth pointing out that neither is predicted to have a particularly successful career in the NFL with this metric in that the combined metric predicts them to be only average, while the bust metric predicts them to bust.  In either case, it's not a ringing endorsement for them to be a Top 13 pick in the NFL and I would even be wary of spending a 1st round pick on either one.  Another interesting name to consider on this list is Brett Smith out of Wyoming.  The combined metric has him as a league average prospect, but his bust metric is appreciably low which gives him very low risk.  I would not spend a first round pick on him though as his success score is pretty low, but as a mid-round sleeper the Vikings could do a lot worse than Brett Smith.

All that said, with only a 73% accuracy rating and 10 names on the list, odds are the metric will miss on about 2-3 of these prospects.  Your guess on which ones it will miss is as good as mine.  If I had to choose quarterbacks that I would definitely want the Vikings to avoid though, it would be Bortles, Manziel, Mettenberger and Garoppolo as their bust potential is just much too high for my tastes.  After Teddy Bridgewater, I'm not particularly fond of any of the quarterbacks in the 2014 draft, and if this metric is to be believed, this looks to be a pretty weak class at the top end, but a decently deep draft in the middle.  If the numbers are to be believed, this draft should produce a fair number of good backup quarterback options and a few league average options, but not many elite, successful quarterbacks.

In This Article

Teams
X
Log In Sign Up

forgot?
Log In Sign Up

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.

Join Daily Norseman

You must be a member of Daily Norseman to participate.

We have our own Community Guidelines at Daily Norseman. You should read them.

Join Daily Norseman

You must be a member of Daily Norseman to participate.

We have our own Community Guidelines at Daily Norseman. You should read them.

Spinner.vc97ec6e

Authenticating

Great!

Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.

tracking_pixel_9341_tracker