Hello everyone! I'm back with another off-season QB article. After showing how each QB did in relation to each other this year, I went back to the 1996 season to see if I could develop a system which predicts whether a QB drafted in the first three rounds would be a hit or a bust. Specifically, can efficiency metrics and a season-by-season analysis of a players college career help determine what is a likely outcome for them when they enter the NFL? To do this I analyzed players the same way I analyzed them in the previous article: comparing them to their peers by normalizing their efficiency metrics, and analyzing their year-to-year stats to look for any signs of progression, regression or a plateau. Without further ado, lets get to it.
To start with, a few 'ground rules' need to be set up and explained. Basically, this method uses purely statistical analysis. There won't be any mention of intangibles or arm strength or hand size etc etc after this paragraph. The reason being is that I essentially co-opted scouts to do all that work for me. Basically, I made the assumption that if scouts/front office people deemed a QB worthy of being selected in the first three rounds, they made the determination that a player had sufficient intangible qualities (such as size, arm strength, leadership, etc.) to merit a high draft pick. The system isn't perfect, but it allows me to ignore all those traits not purely focused on statistics. I think that this is beneficial because I am not a talented film watcher. I don't really know what to look for, nor do I posses the requisite skill to determine that if something is wrong, how easily it can be fixed. With that out of the way, lets move on to the method.
What I did to analyze QBs was similar to how I measured them before: I created a database of their normalized efficiency metrics (completion %, Y/A, AYA, and TD:INT) in order to compare them to each other. I then averaged the list into two separate stats: averaged completion % and AYA, and averaged completion %, Y/A, and TD:INT. Once that was done, I organized the list from greatest to fewest and culled the data down to a reasonable amount. I only accepted QBs whose first average (completion % and AYA) had at least a value of 0.5, with 0.0 being average. So all the QBs on the list were above average that season in relation to their peers.
Once the list was culled down (it generally was about the top 30-35 players in a season who met the 0.5 criteria) I examined the top ten of both the completion % and AYA average, and the completion %, Y/A and TD:INT average. I made the determination that a QB had to at least fall within the top ten of one of those average lists to make the cut. Once they made the initial cut, I looked at them season-by-season to see if there was any progression, regression or a plateau in their college game. If a player was in the top ten, but regressed or plateaued their final year(s) I considered them a miss. If a player made the top ten without regressing or plateauing, I considered them a possible hit. I then went and compared QBs who were drafted in the last 16 years (97-13) in the first three rounds to see how my system fared vs the actual results of QBs drafted in the first three rounds. There were some solid hits and some discouraging misses, but all in all it worked out better than I could have hoped. I'll share the results down below, but first I'm going to first quickly walk you all through the process I described above with pictures, as I have a feeling it will make a lot more sense visually.
First, their raw stats were pasted into excel. For my research, I primarily got my data from sports-reference.com. Once I had the raw stats for a season, they were normalized into a list and averaged, as shown below.
The above screen-shot was from the 2003 college season. You can't see it in the above image, but the list continues on down for a total of 119 QBs. This next image is after I have taken all players who at least meet the 0.5 first average qualification.
As you can see, certain players are highlighted in green or red. The system is pretty self explanatory; if a player is in green, it means the system correctly predicted whether they would be a bust or a hit. If they are in red, the system failed in its prediction. As you can see, for this particular year, I correctly predicted Big Ben and Rivers being successful QBs, while missing on Manning and Schaub. The Schaub prediction is close, as some may consider him a bust while others consider him a success. In the end, I chose to consider him a success. I wanted to be as conservative as possible so as not to taint the results in any way. So if a player was on the edge, they generally went against the system.
If you look into the upper-left hand portion of the second image, you will see a hits and busts section. Its just a simple counter on how I did predicting a QBs success from that year. For this particular year (2003 season, 2004 draft) I hit on 2/4 QBs and correctly predicted the only bust. Now that you (hopefully) have a fairly good understanding of the system lets get to the system-wide results and list some noteworthy successes and failures.
System Wide Results by Round
I decided to break up the results round by round, because I didn't want to give the appearance that I was using the 2nd or third round predictions to buoy the overall statistics.
In the first round, between 1997 and 2013, there were, by my count, 44 players selected. Of those 44, 15 were considered hits and 29 were considered busts. Of the 15 total hits, my system correctly identified 13 of the 15 successful first round QBs.
As you can already conclude from the pictures above, I correctly predicted Rivers and Roetthlisberger as successes. The system also hit on the following players:
- Peyton Manning
- Chad Pennington
- Andrew Luck
- Daunte Culpepper
- Cam Newton
- Alex Smith (who I don't consider a bust, and am happy to debate the point with you in the comments section)
- Aaron Rodgers
Those are just some of the correct predictions. I only missed on two successful first round NFL QBs, one of which you know from the above images is Eli Manning. The other was Matt Ryan. In my opinion, both QBs are overrated, but that doesn't mean they are busts. Manning has won two Super Bowls, and Ryan has been right around league average for his career in terms of efficiency, including leading his team on a deep postseason run last year. It sucks that I missed on both those QBs, but looking at the stats, they both displayed all the tell-tale signs of being crappy NFL QBs. It just didn't work out. In Ryans case, I suspect his deficiencies are somewhat covered up by the pair of all-pro WRs and HOF TE he plays with. With Manning, he only has two seasons of above average efficiency in his entire career, but seems to play well when it matters most: the playoffs. Getting back to the point of the article, so far I'm sitting at 13/15 first round picks, which translates to an 87% mark.
As you can imagine, there are many more busts than hits for NFL QBs, as we are, as Vikings fans, intimately aware of. By my count, there were 29 QBs taken in the first round that were busts. Of these 29, I accurately predicted 21 of these future failures, including:
- Tim Tebow
- Matt Leinart
- Byron Leftwich
- Joey Harrington
- Kyle Boller
- Christian Ponder
- Blaine Gabbert
- Brady Quinn
These are just a few of the correctly predicted busts. Unfortunately, no system is perfect, and there were a few egregious misses. Here a few of the worst offenders
Those players listed above are your veritable poster children for worst top picks in the NFL. In particular, the Leaf, Carr, Couch and Russell misses are the worst of the worst. The reason being is that they all, statistically speaking, appear to be great picks. Russell, Leaf and Carr in particular pass the tests with flying colors. Like I said before, no system is perfect, but those three are some big misses. The reason why its easier to find hits then busts (statistically speaking) is that there are any number of reasons a player might bust that doesn't necessarily have to do with their stats. They may be lazy, or immature, or any number of other issues that wouldn't show up on a stat sheet. This isn't to try and make excuses, only to point out that it is a rare player who is able to pile up great efficiency metrics in college yet be a complete failure in the NFL. Unfortunately, those are also the most notorious busts and thats why they stick out like a sore thumb on this list. The only consolation for me is that the scouts got duped as well. Or Al Davis in Russell's case.
There were far less QBs selected in rounds 2 and 3 off the draft than the first over the past 16 years, so the lists of successes and busts are quite a bit shorter. Here are some of the successful hits on second round QBs.
And that is it. By my count there were only 4 successful NFL QBs drafted in the 2nd round since 1997. I got half of them right, which isn't a great mark but, statistically speaking, much better than teams do themselves when drafting. The two I missed on are some pretty well known names; in fact I bet some of you have already guessed at least one of them. Drew Brees was drafted out of Purdue by the Chargers in the 2nd round, and has been a massive success in his NFL career thus far. The other success that I missed on was Colin Kaepernick. They both regressed between seasons in college, so the system pegged them as busts. Its frustrating to miss on Brees in particular, because he has been so great in his career, particularly since he has gotten to New Orleans.
It may seem a little out of place to say that there were notable busts from the second round, but there were some well-regarded names who fell out of the first round of the draft, only to be picked up in the 2nd round by teams eager to pounce on a player who was rated well but fell for whatever reason. Here are some of the correctly predicted busts.
All told, there were 13 QB busts in the past 16 years. I correctly predicted only 8; still a solid percentage, but not as good as my first round stats. In no particular order, the five misses are as follows.
Its not as bad to miss in the second as the first, but a second round pick is still valuable. In the end, I ended up getting 8/13 busts correct, or 62%. Again, not as good as the first round, but still pretty good numbers overall. Lets check out the third round now.
My success rate for correctly predicting successful NFL QBs falls to its lowest in the study: only 33% of successful NFL QBs were predicted using my system. As it were, there are only, by my count, 3 successful NFL QBs drafted in the third round in the last 16 years. I'm sure many of you can guess the one success, as he was a very recent pick. That player is Russell Wilson. With his efficiency metrics and raw stats, if he was taller he would have been a very high 1st round pick. As it was, the Seahawks got a steal.
The only other two QBs who were successful but weren't predicted were Matt Schaub, which I mentioned above, and another recent draftee, Nick Foles. Both QBs were not in the top ten compared to their peers in their final season before the draft. In Foles case, he was close (16th on the first average) but not quite there. Then again, his 0.89 was the same as Brady's, and Brady missed it by 1 spot. There clearly needs to be some further refinement done to the system; i'm actually thinking about playing around with age as a factor in grading prospects. Some work has already been done in that regard by others. But that is another topic for another post. As it is, Schaub regressed enough, especially compared to his peers from one season to the next that it dropped him from 4th on both averaged Z tables in 2002 to 13th and 24th in 2003. He was fairly close to passing anyway, being only 3 spots out of the top ten in relation to his peers, but he regressed enough in certain areas (AYA and TD:INT) to not pass the regression test anyway. Foles, on the other hand, didn't regress, he actually progressed from year to year, albeit slowly, but he just wasn't quite good enough in relation to his peers in his final season, especially his AYA, Y/A and TD:INT. Below are their respective college stats from year-to-year.
There definitely is room for improvement in this system, but after spending just a few days on it, I've already hit above 70% in both the busts and the hits categories for QBs drafted in the first three rounds. Becaus I was curious about QBs who weren't drafted in the first three rounds but were still hits (Tom Brady comes to mind) I applied the same test on him to see what would Brady look like if he were drafted in the first three rounds instead of the 6th round. Sadly, Brady actually would have failed the test, but only by the slimmest of margins; he finished 11th on one average list and 12th on another. If I had decided to cut off the list at 15 instead of 10, I would have hit on Brady and Schaub. I may yet go back and tinker with the numbers to see what is an equitable place to cut off, and instead of picking the top ten I might go back and study what their averaged Z score needs to be in relation to their peers instead of just an arbitrary cut off. The reason for doing this would be because every QB class is different. Brady's first averaged Z score was 0.89, while in another year, Matt Schaub was 13th at 1.12. Sadly, I most likely won't have the time in the coming weeks (and months) as I will be busy back at school.
Current QB Class
Now that we've gone over the system to see how it works, and how it has worked historically, lets apply it to the current QB class and see what players would register as hits or busts if they were drafted in the first three rounds.
Lets take a look at the predicted hits. Note that, obviously, not all of these players will be hits, as the success rate sits just above 70%, but those are pretty good odds. Here are the projected hits from this years QB class.
My man-crush on Bridgewater knows no bounds, apparently. He ends up in the top 3 in both averaged lists. He's coming off historic levels of season-to-season accuracy. I still feel that he is as close to a sure thing as you will get from this years QB class.
Certainly a very good consolation prize for the team that misses out on Bridgewater, Manziel passes the tests with flying colors. Inside the top ten on both averages list, he has seemingly grown as a passer this year as well. Looks to be a great prospect.
Bortles makes the cut, although he didn't make the top ten in both. Still, his lowest averaged score was 11th, so its not like he missed by a lot. Bortles also seems to be a solid pro prospect.
Boyd scores very well in relation to his peers; if he is taken in the third round or higher, statisitcally speaking he would be a hit. He has progressed every year as a starter and has done fantastically well compared to his peers, finishing at 5th and 9th on the two averaged lists.
Mettenberger passes the test with flying colors as well. He finishes 6th and 8th in relation to his peers and has progressed from year-to-year. What might prevent him from being drafted in the first three rounds is his knee injury, which will keep him from working out for teams.
Game Manger Got Game? That is a link to an article about McCarron from the creator of the Z-Score metric, Matt Freedman. Basically, he asks the question: is McCarron unfairly punished due to the reputation he plays with an all-star team? The conclusion, based on his placement relative to his peers, is that he is indeed a solid prospect, finishing 8th and 11th on the two averages tables. McCarron has regressed from last year however. The question is, it enough to make him fail the system? The way I graded prospects was that they were only allowed to ignore the regression in their game if they were still elite (statistically speaking) year to year. McCarron, for all the hate he seems to garner and his reputation as a game manager aside, passes all the tests. It will be interesting to see how his career shapes out.
Well, we've got 6 hits according to the system. Now, it is highly unlikely that all 6 get drafted inside the first three rounds and are all successful QBs. Wait, did I say 6? I feel that i'm forgetting someone.....
For those of you pining for defense in the first and Garoppolo later on, you seemingly have some confirmation that he would be a solid prospect. He places inside the top ten on both average lists, and seems like he'd be a solid prospect. Small school concerns aside, he was, statistically speaking, very good this past year.
Anyway, like I was saying, it is unlikely that all 7 of these guys get drafted inside the first three rounds. If they somehow ARE all drafted inside the first three rounds, with a hit rate of 73%, figure out of these 7 guys, two would be misses and end up being busts. If I had to guess which two they would be, my money would be on McCarron and Mettenberger. But who knows, for all I know it'll be Bridgewater and Manziel who'll be busts. Speaking of busts....
Poor Derek Carr. He gets lumped in with his brother, and my system seemingly doesn't like him either. He finishes outside the top ten in both categories, but may be a victim of the relative strength of this class, as both his scores were above 1.0. If I go back and reanalyze the system like I talked about earlier, Carr very well may end up being a hit. As it is, he failes for now, but not by much. He ends up 12th and 13th on the two average tables.
Murray seems like a pretty weak prospect in a relatively deep QB class to be honest. He has regressed big time from his previous year, and is pretty poor in relation to his more heralded cohorts. I know some were clammoring for us to wait on QB and grab Murray, but that doesn't seem like a viable option. Granted, he likely won't be drafted in the top three rounds, so if a team were to draft him with a 6th round pick and he doesn't develop, that team wouldn't lose a whole lot (as its only a 6th round pick). Still, don't look for Murray to be the next Tom Brady. Now that i've said that, of course he will.
Fales also seems like a weak prospect, as he ranks even lower than Murray on both average tables. I can't really understand why CBS Sports.com has him rated as a potential 2nd or 3rd round pick.
Another head scratcher here, Smith was just plain not that good this year. He regressed horribly this year, with his AYA and Y/A both dropping over a full point. It seems as if scouts are hoping he will regain his 2012 efficiency, but history is not on their (or his) side.
Well, there you have it. That is the first draft of the system i've come up with to grade QBs. What are your thoughts on it? Brilliant? Total waste of time? Somewhere in between? Let me know what you think I've missed, and who you would like to draft this year at QB.
EDIT: I was asked to post the full list of QBs that were used in the database for 2013. The table should be pretty straightforward, Green are predicted hits while Red are predicted busts. Anyone not color-coded is either not coming out this year (Mariota, Winston) or just aren't serious prospects. For example, Connor Shaw leads one of the lists, but he has almost no hope of being drafted in the first three rounds. And even if he did, he would still be pegged as a bust due to his regression across the board (besides his TD:INT ratio).
If there are anymore questions about this table, past tables of any years, or the method, please don't hesitate to ask. Thanks again for reading everyone!