Tag Archives: Harvard Sports Analysis Collective

On April 28th, 2011, the Jacksonville Jaguars traded two draft picks–the 16th and 49th overall–to the Washington D.C. Football Team and moved up six spots in the first round to select quarterback Blaine Gabbert tenth overall. Using Kevin Meers’ (president of the Harvard Sports Analysis Collective) draft pick approximate value, did this look like a good trade at the time? No.

Meers’ chart, pictured below, uses the career approximate value statistic from Pro Football Reference going back through the last 20-plus drafts to estimate relative pick worth, and answer the question: what is a fair deal when trading draft picks for draft picks?


Using Meer’s valuation, you can see this never looked like a good deal for the Jaguars. On average the tenth overall pick is 39.9 (units1) more than the 16th pick, or 15 percent better. In giving up their second round, 49th overall pick worth 148.4, the Jaguars seemingly overpaid substantially. The difference between how much moving up was worth and what Jacksonville actually gave up–110 units–is worth the 82nd overall pick, in the middle of the third round. The trade would have been reasonable for a top-three pick; but, even just seven spots later down at tenth, the numbers indicate it was a bad idea.

The numbers do not tell everything. Meers’ valuation does not reveal that Gabbert was rushed into starting prematurely after presumed starting quarterback David Gerrard bizarrely left football, or that he played behind a below-average offensive line in each of his three seasons, including the third-worst league-wide his second year (per Pro Football Focus). These factors cannot be forgot. But after three years of awful play from Gabbert, yesterday the Jaguars traded him to the San Francisco 49ers, for the current fourth-to-last pick of the sixth round, worth about 45 points in Meers’ system.2 Ironic how that is the sort of pick the Jaguars should have traded to move up in the first place.

The good news for Jaguars fans is that it is over. Well, almost, as Gabbert will still count $1.807 million against Jacksonville’s salary cap this season in dead money. The good news for 49ers fans is that we have obtained a 24-year-old backup quarterback, whom our organization once thought highly of (and still somewhat does, apparently), for a very late draft pick unlikely to contribute to our current roster, and the well-documented quarterback whisperer Jim Harbaugh is still our head coach. (Yup, gotta’ save the best for last.)

  1. These particular numbers do not mean anything outside the chart; they are used to rank picks in a simple way. They are derived from real numbers, the career approximate value figures of Pro Football Reference. 
  2. The 49ers gave up the current fourth-to-last pick of the sixth round for Gabbert. That pick will not be 189th overall, as the league has a handful of compensatory selections to award between some of the rounds, but it should be close. 

In the 1980s, Dallas Cowboys head coach Jimmy Johnson sought to determine a baseline expected value for every pick in the NFL draft. This was a very good idea. Rather than just winging it each year when considering draft pick trades, expected values would provide the Cowboys front office a framework to build upon. Should Dallas trade the 20th and 80th overall picks for the 5th overall pick? How about the 50th pick for a first round pick next season? These questions require a sound process for good answers. Draft picks are precious commodities; aside from players, they are the currency teams use in deals with each other.

Coach Johnson’s system was not perfect. His extensive knowledge of football through years working in the NFL made the chart a good starting place.1 It contributed to the famous Herschel Walker trade (which has its own Wikipedia page), and helped set up the Cowboys three Super Bowls championships in the 1990s. But Coach Johnson’s preliminary valuation lacked a scientific process. Entire decades have gone by. There are oodles of data on players taken in the NFL draft, the length of their careers, salaries earned, touchdowns scored, tackles, expected points and win probability added, etc.

In the fall of 2011, Kevin Meers, of the Harvard Sports Analysis Collective, revalued NFL draft picks using the Career Approximate Value statistic compiled by Pro Football Reference. (Do read Mr. Meers’ actual article, which details the process.) This system is also presumably not 100 percent perfect, but it is a vast improvement, examining the career output of every single pick from 1980-2005.

How do the two systems compare? These are their respective charts:

Jimmy Johnson’s Draft Pick Value Chart, 1980s

Jimmy Johnson Draft Pick Value Chart

Kevin Meer’s Draft Pick Value Chart, 2011

Kevin Meers Draft Pick Value Chart

Notice first that Johnson’s system is more extreme, valuing the first overall pick at 3,000 and the 224th pick at three.2 Meers’ system values the first pick at 494.6 and the 224th at 39.8. Theoretically, the first pick is probably not worth 1,000 (!) last picks. Even if one of those picks is Tom Brady only one in one hundred times, 1,000 last picks would still yield ten Tom Bradys (Tom Bradies?) while just one first overall pick could yield a maximum of one Tom Brady. Of course, that sentence illustrates that theory is sometimes ridiculous! What would a team do with 1,000 late round draft picks, even over a period of several years? Only 53 people can make the team. Nonetheless, Johnson’s system appears to overvalue the first overall pick by at least a factor of ten, if not a hundred.

The key problem with Johnson’s chart is that it overvalues earlier picks and undervalues later picks. Earlier picks are still better! (Duh.) Meers’ chart, a product of data, still indicates that with each additional pick, the likely career output of the drafted player declines. And Meers’ chart still finds that the rate of decline is decreasing: the drop in expected value from the first to the second pick is many, many times larger than the drop from the 223rd pick to the 224th. The theory and logic behind each chart is the same. Meers’ simply uses actual data, which reflects a different, more accurate picture.

As I wrote yesterday, the San Francsico 49ers are in excellent shape for the upcoming 2014 NFL Draft. Meers’ chart is a big reason why. The 49ers will likely have SIX picks among the first 100 in the draft this spring. That the 49ers first pick comes 30th overall is not a reason to get down in the slightest. The antiquated valuation undervalues such picks: the 100th pick is worth only one-three-hundredth of the first pick in Johnson’s system. But, actual data indicates the 100th pick is more likely worth one-fifth of the first. The 49ers should have six picks with fairly high expected values. And with an already talented roster, they may trade to move up and get a player even more likely to succeed, without giving up all their picks in later rounds.

Remember, draft picks are free money in the NFL; teams get them every year just by being in the league. With the most picks in this draft, the 49ers are rich. Big spending is not just a thing for free agency.

  1. Similar to Dick Vermeil’s Two Point Conversion Chart, which I wrote about here. 
  2. The first overall pick is worth 3,000…what? Dollars? No, just units. The charts have been scaled to simple numerical values, as their purpose is for relatively ranking the picks within them. 

It’s Tuesday, Week 11 is in the books, I lost fantasy football 107.22-105.52 (within the margin of a 17 yard Rob Gronkowski catch, or a Matt Prater field goal, or the Seahawks not getting a return touchdown, etc.), and my 49ers lost two games in a row for the second time under Jim Harbaugh, and the second time this season, and I’m still not sure I’m ready to talk about it. But a couple things Harbaugh did are bothering challenging me quite a bit. A couple challenges, as it were.

With two timeouts and 3:33 remaining in the first quarter of a 0-0 game, Harbaugh challenged that Drew Brees had crossed the line of scrimmage before completing an 8 yard pass to Darren Sproles on a 1st&10 from the 49er 25. Many people, including 49ers beat writer Matt Maiocco, have pointed out that the challenge was terrible, as replays clearly indicated Drew Brees wasn’t close to crossing the line of scrimmage. After the game, Harbaugh confessed “We didn’t have a video review [on that challenge].” Challenging a ruling that was obviously correct is, uh, obviously bad, and revealing that you challenged it without any evidence to the contrary, um, also obviously bad, but I wondered: why challenge this play in the first place? 2nd&10 is harder than 2nd&2, sure, but worth one of your two (or three) challenges and the risk of losing a time out? This isn’t the Jacksonville Jaguars of Week 8, it’s Drew Brees and the Saints of Week 11. Over the course of the game the Saints averaged 5.8 yards per offensive play. Even if Harbaugh had challenged on firmer ground, and the call was reversed… so what? The Saints were already looking at a 42 yard field goal (in a dome), and there were still a few yards between them and the end zone. To save the touchdown, the defense would need a stop either way, and it could come on a new set of downs. To shut them out, the defense would need to create a turnover. Both of these situations are quite possible whether the Saints have 2nd&10 or 2nd&2, so why risk it?

Surprise! The numbers from Brian Burke’s Advanced NFL StatsWin Probability Calculator suggest there is good value in challenging.1 In the Saints’ resulting situation, 2nd&2 from the opponent’s 17 in a 0-0 game with 3:22 left in the first quarter, their probability of ultimately winning was 0.65.2 If Harbaugh had won the challenge, making it 2nd&10 from the 25, the Saints’ win probability would have been 0.61.3 Increasing your chances of winning by 4% isn’t a lot, only actually it kind of is. (You may remember, this game came down to the final play, when the Saints’ Garrett Hartley kicked a game-winning 31 yard field goal.) In an article last month on, Kevin Meers, Co-President of the Harvard Sports Analysis Collective, laid out some neat work he’s done on coaches’ challenges. Using the difference in win probability resulting from winning the challenge minus that resulting from losing the challenge, which he dubs “leverage”, Meers charts all challenges in the 2012 season.

2012 Challenges by Leverage

Neat huh? That first challenge’s leverage was 0.04, and probably worth going after, again forgetting, as Harbaugh did, the replay which guaranteed the original call would stand. But you know, if what Harbaugh thought had happened had actually happened, it would have been an okay move. And that’s actually more than many NFL coaches can say. 15% of all challenges last season had zero leverage– whether they were successful or not had no discernible impact on the outcome of the game. In that article Meers also attempts to value timeouts, gauging them to be worth around 0.03 in win probability. I won’t get into his methodology (though I may try to build on his foundation in the coming weeks), and you can read it for yourself, but just take that 0.03 number for a moment. In terms of the lost timeout, it’s worth challenging when you yield 0.04 in win probability if you win the challenge 42.9% of the time. While I wouldn’t take that 0.03 number as a universal truth, and it’s worth mentioning again that these win probabilities are averages not personally tailored to a team’s own defense or offense, it’s still interesting. Mostly, it’s suggestive of the cool things we can learn once these types of models are further refined and improved.

Harbaugh’s second challenge was a similar story. Down 7-0 with 10:21 left in the second quarter, Harbaugh challenged that Kaepernick’s pass to Jon Baldwin in the end zone on 1st and 10 from the Saints’ 11, ruled incomplete, was in fact a touchdown. This was the ole’ Calvin Johnson rule about securing the ball for 547 minutes after you make a catch and go out of bounds, and while stupid, is a rule that coaches and players (and fans) know about. Under the rule, it was clearly an incomplete pass, leaving the 49ers with 2nd and 10. This was a little more troubling than the first failed challenge. Harbaugh said of the decision to challenge the play “I was talking to Eric [Mangini, our challenge consultant up in the booth]”. Cleveland Browns fans surely won’t be surprised by Mangini’s involvement, as he, uh, never really panned out as their head coach. Unfortunately he seems to have brought some similar failings to his new post. But failings aside, how often would the 49ers need to win this challenge for it to be worthwhile? 30% of the time.4 Given the replay (and the fact the game was in New Orleans), I’d say the chances of an overturn were zero, maybe 10% being generous. Nonetheless, that wasn’t Harbaugh’s area. Hearing the false possibility from Mangini, and not knowing the 49ers would score on the next play, this Harbaugh challenge was much more defensible.

  1. Assuming the challenges themselves aren’t completely hopeless. That’s still on Harbaugh. Well, actually the truth may be more complicated. More on that later. 
  2. Also, their probability of gaining a first down was 0.75; a field goal 0.41; and a touchdown also 0.41. 
  3. First down probability 0.52, field goal 0.37, touchdown 0.34. 
  4. The 49ers had a 0.48 probability of winning if they won the challenge, and a 0.41 probability if they lost, for a challenge leverage of 0.07. 
%d bloggers like this: