AP Top 25 College Football Poll and Voter Rankings
Not every AP Top 25 voter is created equally. In this article we’ll study how one can come up with voter rankings to determine who the good and the bad pollsters are. Regardless of individual quality, the “wisdom of the masses” approach employed by the AP top 25 poll tends to be pretty good at comparing the relative quality of the top college football teams. Our goal is to see whether or not giving more weight to the ballots of the more accurate voter’s results in more accurate rankings.
Determining Voter Rankings
First, we need to discuss how we rank the various pollsters that vote on the AP top 25 college football poll. Voter rankings necessarily need to depend on accuracy, but the relatively small number of top-25 matchups results in too few matchups to properly determine whose rankings are best.
To solve this problem, we use the end-of-season rankings as the undisputed truth, as gospel, for the ranking of the top 25 teams. Then, we compare each voter’s week by week rankings to the aggregate end-of-season rankings to see how close they were to perfectly accurate throughout the season.
If my weekly rankings closely resemble the final rankings, then I did a pretty good job of determining who the best teams were. If my weekly rankings do not resemble the final rankings, then perhaps I was not as good of a judge of quality as other pollsters were. Highly accurate pollsters are those whose week-by-week rankings closely resemble the final rankings. If your rankings look like the final rankings earlier on in the season, you are awarded for having accurate foresight.
The specific mathematics…for an individual ballot by an individual pollster, we add up the total error between that pollster’s weekly rankings and the end of season rankings. Here is a specific example. Alabama finished the 2019 season number 8 in the aggregate AP poll. Many pollsters had Alabama #1 in their pre-season rankings. Those who did had an ‘absolute error’ in predicting Alabama’s final rankings of 7 spots. If you were more bearish on the Crimson Tide and had them ranked 4th in your pre-season rankings, you would only accrue |7-4|=3 points of error.
In this exact way, we go through each team in each ballot and add up the total error in predicting that team’s final ranking. The AP top 25 voters who had smaller errors are determined to be the most accurate.
After doing this, we are able to determine our AP top 25 voter rankings and can answer the question: “Which AP voters are most accurate in their college football rankings?”
2019 AP Top 25 Voter Rankings
Not every voter is obligated to submit a ballot every week. The list of voters we consider in 2019 are those that missed at most 5 weeks of submissions. For a full list, see this excellent resource. We start by going through each and every ballot and determining who was most accurate each week during the 2019 season. The week-by-week most accurate pollsters are shown in the graphic below.
To figure out the season-long most accurate voter, we need to add up each voter’s error over the course of the season. Weekly consistency is key. For each pollster we computed their overall accuracy by combining their weekly accuracies after normalizing them for the average errors in a given week. Why do we normalize? Naturally, the rankings of every pollster at the end of the season will be highly reminiscent of the final AP top 25 rankings. Towards the beginning of the season, the errors will be larger. Normalizing makes sure that each week matters the same amount when combined into an aggregate ranking of AP voter accuracy.
Below are the top 5 AP voters for the entire 2019 college football season.
*Matt Brown is now a senior editor The Athletic College Football, Sports on Earth is defunct
How much more accurate were these voters than their counterparts? The difference between the best and the worst voters in the 2019 AP Poll was an average error of just over 30. That means that for every team in the top 25, Neil Ostrout and co. put that team about 1.5 spots closer to their final ranking than the worst pollsters did. It doesn’t necessarily sound like that much, but when you realize just how similar everyone’s top 5-10 teams are, it puts that difference into perspective.
Now, having sufficiently shouted out the 2019 best AP top 25 voters, we turn our attention towards using this information to potentially generate more accurate weekly rankings.
Weighting the AP Top 25 Poll by Voter Rankings
What we want to show is that by giving certain AP voters more impact, our rankings will converge to the ‘end-of-season’ rankings much more quickly. That is, when we weight the good voters’ ballots more heavily, the in-season polls will more accurately reflect the end-of-season poll than if we give everyone’s vote the same impact.
However, we must slightly alter our method of determining the most accurate pollsters in this section. If we want to reweight, for example, the week 4 poll, we need to be careful to only use information that would have been available to us in week 4. In particular, we can’t use end-of-season rankings to determine the most accurate voters in week 4 because during week 4 we don’t know the end-of-season rankings.
So, in order to rank the AP top 25 voters at week 4 in the season, we perform the exact analysis of the previous section but use week 4 as our rankings to compare to. At that point in the season, the week 4 rankings act as our best guess of the overall rankings of the teams so those pollsters whose ballots look most like the week 4 rankings are deemed the most accurate.
Then, we compute the week 4 rankings by taking all the week 4 ballots and weighting the more accurate pollsters ballots more heavily.
In this way, we computed a re-weighted AP top 25 poll for weeks 2 through 17 by assigning extra weight to the accurate voters in our voter rankings. Then, we finally come to the important question: how much better are our weighted AP top 25 rankings versus normal AP top 25 rankings.
It turns out….not hardly at all.
That’s pretty unsatisfying right? Why did I write an entire article about something that doesn’t really matter? (I can’t begin to tell you how many times I’ve written reports, worked on projects, etc. where my only result was to warn others not to go down the same track unless they want to waste months.)
But a negative result is still important. The whole point is that if someone is in-tune enough with the sports world to get a ballot in the first place, their guess at the top 25 college football teams is probably just as good as anyone else’s. There is enough luck in sports that no matter what you do or how much you know you can’t be perfectly accurate.
The most accurate voters deserve to still be recognized – and I intend on doing that at the end of every college football and basketball season – but the difference between the best and the worst voters is within the normal variation one might expect due to chance.
Still, congrats to Neil Ostrout and the Journal Inquirer for having the most accurate college football polls in 2019.
To subscribe to future articles and updates, please use the form below!
