NFL and AWS unveil new 4th down metric to judge whether your favorite coach made the right call
For the 2021-2022 NFL season that started on Thursday, the NFL and AWS are partnering to add a few new statistics to their Next Gen Stats tool.
The league will now track 4th down and two-point conversion analytics with Next Gen Stats Decision Guide powered by AWS.
Fourth downs continue to be some of the toughest -- and most contentious -- decisions leaders of the game have to make on Sundays, and the NFL will now give fans some insight into their choices with the help of AWS and tracking technology.
Matt Swensson, vice president of Next Gen Stats at the NFL, told ZDNet that the Next Gen Stats Decision Guide is built on a series of machine learning models using Amazon SageMaker to power live 4th down and 2-point conversion decision analytics.
He explained that the decision equation focuses on two main components: win probability, which informs how much the game will change in the hypothetical event of each outcome, and conversion probability, which tells us the likelihood of the offense converting a fourth down or 2-point conversion.
"The 4th down decision guide is an interesting one in that it works by using other models. We had base level stats that we derived information from; then, we started creating models out of the derived stats. Now we're creating more metrics using more models, so it's just layer upon layer," Swensson said.
"We take a combination of the win probability at that point in the game for a team and then a conversion probability on whether they are able to convert on a play. You basically can then put probabilities against questions like 'Should they go for it?' 'Should they punt?' 'Are they in field goal range?' etc."
A screenshot of what the NFL's 4th down metrics look like.
The statistics aren't for coaches but rather for NFL Network, broadcast partners and fans of the game interested in learning about how decisions by coaches are made.
For some games, broadcasters will ask the NFL for statistics or probabilities, and the league will share data they have run through AWS.
"Coaches are still going with their Instinct and their decision card that they have on the sidelines, but what we've done is started to quantify that and educate the fan a little bit on why you would make that decision and why you may choose actually to go against what the odds are favoring," Swensson added.
Swensson told ZDNet that what interested him the most was seeing the analysis of how different coaches approach 4th down and how they have dealt with the decision going back to 2016.
Some coaches are more willing to take risks, while others are happy to punt and see how their defense responds. The NFL examined how teams performed on 4th down over the opening weekend's slate of games, detailing a number of interesting findings in a blog post on Monday.
The NFL found that the decision of Dallas Cowboys head coach Mike McCarthy to kick a 21-yard field goal on 4th-and-3 midway through the third quarter from the Tampa Bay 3-yard was "among the most costly decisions of Week 1."
The decision cost the Cowboys 5.8-percent in net win probability, the most value lost on a 4th down decision entering Monday Night Football according to the Next Gen Stats Decision Guide. The Cowboys' odds of converting were 51%, the NFL found.
Unfortunately for Cowboys fans, that wasn't the only blunder of the night caught by the Next Gen Stats Decision Guide. McCarthy decided to kick a field goal in a 4th-and-3 situation with 6:41 in the second quarter from the Buccaneers 13-yard line. While that decision was less egregious according to the data, it still had an effect on the Cowboys' eventual 2-point loss.
The NFL noted that McCarthy made another tough 4th down call with 1:29 remaining in the game, electing to kick a field goal from the Tampa Bay 30-yard line.
"Their win probability after making a field goal would still be only 44%. If they went for it on 4th-and-6 (a 37% proposition) and converted, the Cowboys would have had a 69% chance to win the game. The Cowboys elected to kick a field goal -- which our model rated as a true go-field goal toss up -- followed by a Brady game-winning drive to set up a Ryan Succop game-winning 36-yard field goal," the NFL explained.
On the opposite end of the spectrum, the aggressive play calling of Lions head coach Dan Campbell helped keep his team in the game even when it looked like it might be over.
The Next Gen Stats Decision Guide found that Campbell made the right call on almost all 10 of his 4th down decisions, missing just once on a toss-up 0.1-percent go-for-it recommendation on 4th and 14 from the SF 47 down 21 points in the 3rd quarter.
Campbell repeatedly elected to go for it on 4th down, knowing his team was an underdog and extremely disadvantaged. Campbell was undeterred despite unsuccessful plays on previous 4th downs.
Browns head coach Kevin Stefanski was similarly brazen on 4th down after a 2020 season where he was rated as "the most optimal decision-maker" by the NFL.
"Facing a 4th-and-3 from the Kansas City 15-yard line, the Browns elected to go for it when our numbers said field goal was the most optimal decision (by 1-percent). A difference of only 1% is more of a toss-up decision than a true recommendation. The Browns converted the first down on a Baker Mayfield-Austin Hooper 5-yard completion," NFL researchers found.
"While the numbers say field goal, the difference between the decisions was marginal. The Browns would go on to score a TD two plays later, and after drawing an encroachment penalty on their extra point attempt, Stefanski sharply accepted the penalty and went for two (successfully). According to our two-point model, Stefanski and Co. should go for two if the conversion probability was greater than 49-percent. In this case, the number jumped from 53%to 65%."
One of the most interesting games featuring tough 4th down calls was the match between the Cincinnati Bengals and Minnesota Vikings.
Bengals head coach Zac Taylor was conservative to end the game, deciding to punt on 4th down with 1:55 remaining and the Bengals leading 24-21.
The decision proved to be the wrong one, with the Vikings marching down the field to kick a field goal and tie the game, sending it into overtime.
But Taylor redeemed himself in overtime. With just 0:39 remaining and the game tied, the Bengals faced a tough fourth-and-1 from their own 48. Taylor elected to go for it to avoid the tie, and the team converted.
"Interestingly, in this specific case, given the risk of a failed go-for-it attempt, the model actually favored a punt by 2.2%. If the Bengals converted -- as they did -- their odds of winning jumped to 59%. If they would have failed? The odds would have dropped to 40%," the NFL found.
"Taylor opted against a tie and made them go for it call, paving the way for a game-winning kick. If there was a pattern in Taylor's decision-making, it was this: go for it in fourth-and-1 situations. Out of 11 fourth-down situations, three were fourth-and-1, and the Bengals kept the offense on the field in all three. On the other eight, they kicked. Our model finds most 1-yard-to-go situations are go-for-it scenarios, though it does depend on the situation."
When Next Gen Stats was first launched in 2015-2016, Swensson said that they were initially tracking basic metrics like how fast and far the players were running.
They slowly upgraded to more difficult stats like separation at the time of catch and how much space the offensive line gave the quarterback. But things changed when they tried to figure out whether a defense was blitzing.
"It was a lot of if-then logic, and it was kludgy but worked. We realized that there had to be a much more elegant way to identify certain things. The stat that kicked things into gear and started the work with AWS, especially in the AI and the ML space, was our completion probability metric, which was the first one that really took a bunch of data and used Amazon SageMaker to train a model," Swensson explained.
"That stat started as a spreadsheet where we were trying to tweak some parameters, and I told the researchers ', I think this is something that's really more suited for machine learning.' So we enlisted help from AWS on a couple of fronts, and since then, we've been creating more and more ML base stats, where we take a bunch of data that we've got tagged or labeled."
The NFL has worked with AWS since 2018 and unveiled other new statistics this year, including Quarterback Expected Rushing Yards, Quarterback Dropback Type, Next Gen Stats Big Play Score, and Expected Fantasy Points.
For the future, Swensson said there are plans to do more data dives on the defensive side of the ball in identifying coverage schemes and more. There are also plans to get involved in the fantasy football space using data from the league.
"AWS has been a large reason we've been able to really get to the next step in these statistics. I'm looking forward to our continued evolution with them, and we're always looking for new ways to explain the game to fans," Swensson said.