Average math scores for the class of 2011 fell one point to 514 and scores on the critical reading section fell two points to 489.
Experts say the scores may be falling because a growing number of test-takers are from households where English is not the first language.
The College Board said the results reflect the record size of test-takers. As more students aim for college and take the exam, it tends to drag down average scores.
Meanwhile, other tests taken by more representative groups of high school students have shown reading skills holding steadier.
Still, it's just the second time in the last two decades reading scores have fallen as much in a single year. And reading scores are now notably lower than as recently as 2005, when the average was 508.
The Associated Press contributed to this report.
