Numerous quarterback evaluation metrics exist, but they all drop the ball in one area—isolating the specific variables a quarterback must evaluate before a passing play and assessing whether the quarterback made the optimal decision after the play. To close this gap, the Next Gen Stats (NGS) team used Amazon Web Services (AWS) offerings to develop Passing Score, a new artificial intelligence (AI) quarterback measurement tool that evaluates passing performance.

Over the course of an NFL game, quarterbacks do far more than throw a few passes. They make hundreds of split-second decisions. As they approach the line of scrimmage, they assess factors like the defensive players’ lineup, time remaining in the game, and yards needed to make a first down. The variability of a play’s success significantly rises in complexity once the play starts and the quarterback must make a snap judgement on where to throw the ball. To compound the variabilities, a receiver could make an incredible play on an overthrown ball, or a defensive player might tip a perfect pass up in the air that triggers an interception. In the past, regardless of the play’s outcome, the statistical credit or blame was assigned to the quarterback.

Innovative feature to launch on select devices in the U.S. on November 18, when the New England Patriots take on the Atlanta Falcons on Prime Video.

Now, NFL fans have Passing Score. NGS spent nearly a year solving a series of complex technology problems to create the Passing Score tool. NGS had no existing playbook to follow, but they knew AWS is uniquely positioned to help tackle these challenges with its existing data-driven solutions for sports and deep technology partnership with the NFL.

Building on years of creating and deploying AWS-powered machine learning (ML) models, the NGS engineering and analytics team worked with the AWS Professional Services data science group to optimize and combine seven ML models, including a new model to predict the value of a pass before the ball is thrown. The combined ML models power the new NGS Passing Score. Each individual model provides compelling insights into what’s happening on the field, like predicted yards and completion probability. By combining the models, a score (a number between 50–99) better reflects the intuition used by fans, coaches, and players to assess a quarterback’s passing performance.

Ranking the 2021 QB performances

The 2021 NFL regular season delivered some memorable quarterback performances. We evaluated the regular season’s gameplay and combined that analysis with more than 16 million video frames of player data across the last three seasons of play to answer the question, “Who was the best 2021 regular-season quarterback?”

An illustrated image that says "Ranking the 2021 NFL playoff quarterbacks"
An illustrated image show the two quarter backs with the highest passing scores. In the number one spot is Aaron Rodgers from the Green Bay Packers with a Passing Score of 92. Below his ranking, there is text that says "It is no coincidence Rodgers finished the season as the No. 1 ranked QB by the NGS Passing Score (92), and the Packers earned the No. 1 seed in the NFC for the second consecutive season (99 in 2020)."

The quarterback with the second-highest score is Joe Burrow from the Cincinnati Bengals with a score of 91. Below his ranking is text that reads: "Burrow's last two games of the season (Week 16 vs. BAL and Week 17 at KC) go down as his top two NGS Passing Scores of his career (91 and 90). He scored 80 or above in 14 of 16 games."
An illustration showing the third and fourth quarterbacks with the highest passing scores. 

In #3 with 91 passing score, Matthew Stafford is one of only two QBs to finish the regular season with a max-99 Passing Score against the blitz (the other QB is Patrick Mahomes).

In #4 with a passing score of 89, Ryan Tannehill ranks among the top five QBs in three NGS split categories: inside the tackle box (89), quick (88), and out-breaking routes (84).
In the #5 spot is Patrick Mahomes with a score of 89. Mahomes ranks first among qualifying quarterbacks this season in every situation you'd expect: against the blitz (99), on third down, on the run (92), and targeting out-breaking routes (90).

In #6 spot is Josh Allen with a score of 88. Allen threw an NFL-most 19 touchdowns targeting receivers in the intermediate part of the field (between 10–19 air yards), five more than the next closest QB, helping him earn a max-99 Intermediate Passing Score.
#7 is Tom Brady with the Tampa Bay Buccaneers. His score is 88. Splits where Brady ranked among the top five QBs this season: vertical routes (99, first), third down (96, fourth), in-breaking routes (95, fifth), inside the tackle box (88, fifth), and quick passes (86, fifth).

#8 is Dak Prescott with the Dallas Cowboys. His score is 87. Prescott's Week 18 performance against the Eagles (97 Passing Game Score) goes down as the highest score in a game over the last four seasons.
#9 is Kyler Murray from the Arizona Cardinals with a passing score of 87. Murray's 89 Passing Score under pressure this season (second among QBs) correlates with an NFL-best +11.8% completion percentage over expected (also among qualified QBs).

#10 is  Jimmy Garoppolo with the San Francisco 49ers  with a passing score of 87. Garoppolo took full advantage of Kyle Shanahan's play-action heavy passing, averaging 10.5 yards per attempt on play action (the only QB over 10), helping him earn the top Play Action Passing Score among QBs this season (97).
#11 is Jalen Hurts from the Philadelphia Eagles with a score of 84. Hurts has been among the most dangerous passers on drop-backs of longer than 4 seconds this season (94 Extended Score, second among QBs). He’s less effective as a quick passer (69 Quick Score, thirtieth)

#12 is Derek Carr from the Las Vegas Raiders with a passing score of 83. Raw CPOE, Carr ranks fifth among qualified QBs this season (+2.2%). When adjusting for dropped passes, Carr's CPOE feature jumps to second among QBs.
#13 is Mac Jones from the New England Patriots with a passing score of 83. Jones, the fifteenth-overall pick in last year's draft and fourth QB taken, was the only rookie QB to finish the regular season with a positive CPOE (+1.2%, eighth) and positive pass EPA (+72.0, sixteenth).

#14 is Ben Roethlisberger from the Pittsburgh Steelers with a passing score of 70. Roethlisberger (70 Passing Score) is the only quarterback over the last four seasons (since 2018) to lead his team to the postseason with a score below 82.

Building a winning combination

The NGS Passing Score is a simple representation of a complex equation. By design, it does not simply award all passing yards, touchdowns, and interceptions to the quarterback. Instead, it isolates the contribution of the passer on the outcome of a pass play, agnostic of the receiver. It also controls for the level of difficulty of every pass, to best encompass the traits of a passer that are most predictive of future performance. Finally, it rolls all this into a single metric.

Passing Score works by analyzing tracking data to predict how many yards a receiver will gain if targeted, combined with an enhancement to the completion probability model, which can now also estimate the probability of an interception. The new system created with the AWS team powers the “expectation” metrics used in the primary component of the score.

With this novel AWS-powered methodology now in place, NGS Passing Score can expand and factor in other dimensions like rushing, sacks, and pressures. To learn more about the new metric, check out this blog the NFL created to show you how it all plays out on the field.