It’s Tuesday, Week 11 is in the books, I lost fantasy football 107.22-105.52 (within the margin of a 17 yard Rob Gronkowski catch, or a Matt Prater field goal, or the Seahawks not getting a return touchdown, etc.), and my 49ers lost two games in a row for the second time under Jim Harbaugh, and the second time this season, and I’m still not sure I’m ready to talk about it. But a couple things Harbaugh did are
bothering challenging me quite a bit. A couple challenges, as it were.
With two timeouts and 3:33 remaining in the first quarter of a 0-0 game, Harbaugh challenged that Drew Brees had crossed the line of scrimmage before completing an 8 yard pass to Darren Sproles on a 1st&10 from the 49er 25. Many people, including 49ers beat writer Matt Maiocco, have pointed out that the challenge was terrible, as replays clearly indicated Drew Brees wasn’t close to crossing the line of scrimmage. After the game, Harbaugh confessed “We didn’t have a video review [on that challenge].” Challenging a ruling that was obviously correct is, uh, obviously bad, and revealing that you challenged it without any evidence to the contrary, um, also obviously bad, but I wondered: why challenge this play in the first place? 2nd&10 is harder than 2nd&2, sure, but worth one of your two (or three) challenges and the risk of losing a time out? This isn’t the Jacksonville Jaguars of Week 8, it’s Drew Brees and the Saints of Week 11. Over the course of the game the Saints averaged 5.8 yards per offensive play. Even if Harbaugh had challenged on firmer ground, and the call was reversed… so what? The Saints were already looking at a 42 yard field goal (in a dome), and there were still a few yards between them and the end zone. To save the touchdown, the defense would need a stop either way, and it could come on a new set of downs. To shut them out, the defense would need to create a turnover. Both of these situations are quite possible whether the Saints have 2nd&10 or 2nd&2, so why risk it?
Surprise! The numbers from Brian Burke’s Advanced NFL Stats‘ Win Probability Calculator suggest there is good value in challenging.1 In the Saints’ resulting situation, 2nd&2 from the opponent’s 17 in a 0-0 game with 3:22 left in the first quarter, their probability of ultimately winning was 0.65.2 If Harbaugh had won the challenge, making it 2nd&10 from the 25, the Saints’ win probability would have been 0.61.3 Increasing your chances of winning by 4% isn’t a lot, only actually it kind of is. (You may remember, this game came down to the final play, when the Saints’ Garrett Hartley kicked a game-winning 31 yard field goal.) In an article last month on AdvancedNFLStats.com, Kevin Meers, Co-President of the Harvard Sports Analysis Collective, laid out some neat work he’s done on coaches’ challenges. Using the difference in win probability resulting from winning the challenge minus that resulting from losing the challenge, which he dubs “leverage”, Meers charts all challenges in the 2012 season.
Neat huh? That first challenge’s leverage was 0.04, and probably worth going after, again forgetting, as Harbaugh did, the replay which guaranteed the original call would stand. But you know, if what Harbaugh thought had happened had actually happened, it would have been an okay move. And that’s actually more than many NFL coaches can say. 15% of all challenges last season had zero leverage– whether they were successful or not had no discernible impact on the outcome of the game. In that article Meers also attempts to value timeouts, gauging them to be worth around 0.03 in win probability. I won’t get into his methodology (though I may try to build on his foundation in the coming weeks), and you can read it for yourself, but just take that 0.03 number for a moment. In terms of the lost timeout, it’s worth challenging when you yield 0.04 in win probability if you win the challenge 42.9% of the time. While I wouldn’t take that 0.03 number as a universal truth, and it’s worth mentioning again that these win probabilities are averages not personally tailored to a team’s own defense or offense, it’s still interesting. Mostly, it’s suggestive of the cool things we can learn once these types of models are further refined and improved.
Harbaugh’s second challenge was a similar story. Down 7-0 with 10:21 left in the second quarter, Harbaugh challenged that Kaepernick’s pass to Jon Baldwin in the end zone on 1st and 10 from the Saints’ 11, ruled incomplete, was in fact a touchdown. This was the ole’ Calvin Johnson rule about securing the ball for 547 minutes after you make a catch and go out of bounds, and while stupid, is a rule that coaches and players (and fans) know about. Under the rule, it was clearly an incomplete pass, leaving the 49ers with 2nd and 10. This was a little more troubling than the first failed challenge. Harbaugh said of the decision to challenge the play “I was talking to Eric [Mangini, our challenge consultant up in the booth]”. Cleveland Browns fans surely won’t be surprised by Mangini’s involvement, as he, uh, never really panned out as their head coach. Unfortunately he seems to have brought some similar failings to his new post. But failings aside, how often would the 49ers need to win this challenge for it to be worthwhile? 30% of the time.4 Given the replay (and the fact the game was in New Orleans), I’d say the chances of an overturn were zero, maybe 10% being generous. Nonetheless, that wasn’t Harbaugh’s area. Hearing the false possibility from Mangini, and not knowing the 49ers would score on the next play, this Harbaugh challenge was much more defensible.
- Assuming the challenges themselves aren’t completely hopeless. That’s still on Harbaugh. Well, actually the truth may be more complicated. More on that later. ↩
- Also, their probability of gaining a first down was 0.75; a field goal 0.41; and a touchdown also 0.41. ↩
- First down probability 0.52, field goal 0.37, touchdown 0.34. ↩
- The 49ers had a 0.48 probability of winning if they won the challenge, and a 0.41 probability if they lost, for a challenge leverage of 0.07. ↩