In the months leading up to the recent war with Iraq, President Bush and other conservatives invoked standardized test scores to suggest that American students do not know enough U.S. history to understand the crisis. In fact, as the example of the National Assessment of Educational Progress U.S. History exam indicates, structural flaws and problems with question design make any standardized history test a bad measure of student understanding. Worse still, because the framework of this exam limits test subject areas to a narrow consensus, it defeats the purpose of learning historical knowledge to promote informed democracy and encourages the misuse of history for partisan political purposes.
On September 17, 2002, President Bush introduced initiatives to improve the teaching of history and civics in American classrooms. “Our Founders believed the study of history and citizenship should be at the core of every American’s education,” he declared in his remarks that day. “Yet today, our children have large and disturbing gaps in their knowledge of history.” In order to make this case, Bush relied on standardized tests. “Recent studies tell us that nearly one in five high school seniors think that Germany was an ally of the United States in World War II,” he explained. “Twenty-eight percent of eighth graders do not know the reason why the Civil War was fought. One-third of fourth graders do not know what it means to “pledge allegiance to the flag.”” He then explained the significance of this failure, “Ignorance of American history and civics weakens our sense of citizenship. To be an American is not just a matter of blood or birth; we are bound by ideals, and our children must know those ideals [Bush, 2002].”2
Certainly, it is a truism that an informed citizenry is essential to a properly functioning democracy. However, the failure of students to perform well on standardized history tests is not a sign that our democracy is crumbling. Unlike math or reading, which require specific skills that can be tested for, American history covers an enormous body of material. tests can only examine whether students know a limited amount of specific information. By definition, the facts covered on any one test constitute only a tiny portion of American history in its entirety. This means students can learn an enormous amount of history and none of it might be covered on a standardized Test. Furthermore, standardized tests cannot measure whether students have developed the critical thinking skills necessary to make historical knowledge a means by which they can become well-informed citizens. For this reason, reliance on these tests in order to make educational policy decisions reflects an approach to history that will harm democracy rather than help it.
There are general arguments that could be made with respect to the faults of any standardized test: Teachers teach to the test; the tests discriminate against poor school districts; students have little incentive to perform well, etc. A large, well-organized movement opposed to standardized testing (or at least standardized testing as it is currently practiced in America’s schools) has made these arguments many times. I will not be making those arguments here. Instead, I will be focusing on design aspects of standardized testing and example questions that are particularly bad measurement tools for determining student knowledge or engagement with history. Good teachers can teach many aspects of this subject, but if the material they cover is not on the test, it will appear that student performance is poor. In fact, if teachers teach any subject in-depth so as to make critical thinking possible, their students will have less time to learn specific facts that might show up on such exams.
If standardized history tests don’t measure skills needed to make students active participants in American democracy, why use them? If you believe that there are certain facts that all students must know about American history in order to be good citizens, then these tests can see whether students have learned them. However, no two teachers, schools or school districts will be able to come to a consensus on exactly what this body of knowledge is because there is so much American history from which they can choose. Test designers, in their effort to design questions that are as relevant as possible, approximate an unattainable consensus by creating questions that legitimate established social, political and economic authorities. These questions tend to stress the importance of institutions over individuals, conservative actors over reformers and agreement over dissent. In other words, they tend to reflect conservative political values.
Some standardized test questions cover material which might have been considered progressive in days gone by (for example, Martin Luther King’s “I Have a Dream” speech), but have since been appropriated into a consensus narrative of United States history structured around continual forward progress. And by defining the American history that students ought to know in the most uplifting terms, conservatives, including President Bush, can then use history to promote their political agenda. Under the cover of promoting citizenship, they want to limit American history to information that reinforces their point of view.
Rather than recognize the inherent flaws of creating tests that cover only a small portion of an infinite historical landscape, conservative politicians and educational reformers exploit standardized test scores to promote the idea that a crisis exists in history and civics education. They insist that the house is on fire so that they can sell their version of fire prevention. Teaching conservative values is how they want to stop historical and civic ignorance from reoccurring, and by using standardized tests they can claim their methods are scientifically measured and objective. However, if you recognize how long complaints about the historical stupidity of American students have been around it is very difficult to become alarmed. Sam Wineburg, at the beginning of his book Historical Thinking and Other Unnatural Acts, offers nearly identical complaints about ignorance of American history going back to 1917 [Wineburg, p. viii]. The crisis over the failure of American students to learn the component knowledge of an unattainable consensus is nothing but a political smokescreen.
As evidence for these arguments, I will focus specifically on the National Assessment of Educational Progress test (or NAEP, better known as The Nation’s Report Card), the source for many of the disturbing statistics used in recent years to deplore the knowledge of American students. The NAEP is a series of tests in eight subject areas given to select public and private school students in grades 4, 8 and 12.3 It is conducted under the auspices of the federal Department of Education’s National Center for Educational Statistics and run by the National Assessment Governing Board [NAEP Framework, p. 2]. The purpose of the test is to set benchmarks, as well as assess and track student performance over time. By doing so, educators and lawmakers are supposed to be able to judge the effectiveness of state educational policies [NAEP Parental Handbook, p. 2].
The NAEP program dates from 1969, but the U.S. History test was not given until 1986.4 It was most recently administered in 2001. The areas of American history that are tested upon as well as the specific questions are determined on the basis of an elaborate framework set up by educators, experts and the public [NAEP Framework, pp. 3, 4-9]. The test’s designers say they want students to do more than exhibit their factual knowledge. According to the framework for the exam, students “should have analytic powers to make informed judgements and reasoned decisions [NAEP Framework, p. 13].” This explains why the NAEP history test includes both multiple choice and short-answer questions. The test designers recommend that students spend no more than half the time on the multiple choice questions [NAEP Achievement Levels, p. 6].
The inclusion of open-ended questions is not the only way that the U.S. History NAEP compares favorably to other standardized tests. It is impossible to read the literature that the Department of Education puts out about the NAEP and not be impressed with the care that goes into designing this test. It is both sensitive to the complexity of history and it even reflects many of the recent trends in historical scholarship. However, it is precisely the care that went into designing the test and the sophistication they want students to show that makes the actual test so damning. Despite cultural sensitivity and the desire to encourage sophisticated analysis, the NAEP U.S. History test exhibits structural flaws inherent in any standardized history test. These flaws make it a poor measure of student learning. Furthermore, the last two versions of the NAEP included questions that exhibit extraordinary ignorance of the subject the test is designed to evaluate.
Before getting to these flaws, it is worth considering a trait peculiar to this NAEP test which conservatives have exploited. Unlike other recent requirements for universal public school testing, the U.S. History NAEP is administered to only a sample of the student population at each of the three grade levels it covers. Approximately 29,000 students took the NAEP U.S. History test in 2001 out of a school age population in these three grades of approximately 10.5 million. This means that slightly more than .2% of eligible students took the test.5
Of course, the sample chosen to take the test is designed to be representative of the school age population, but any sample taken at this percentage will have some degree of error in it. The problem here is not that they are sampling (although Republicans had huge problems accepting the concept of sampling when it came to the 2000 Census [See, for example, Vojeda, 1998]), but that the results of the test are never reported as a sample. Bush, for instance, in his September 2002 speech spoke only of “Twenty-eight percent of eighth graders,” not twenty-eight percent of eighth graders tested. Conservatives do not acknowledge any degree of error when reporting NAEP results because that would undermine their rhetoric of crisis. As far as I can tell from the NAEP web site, the Department of Education doesn’t acknowledge any degree of error either.
However, even if you accept the sample as a perfect representation of the student population, the abstract nature of history makes this a bad performance measure for any school. Despite all the care that goes into its design, no test can encompass the infinite diversity of potential historical subjects. Granted, the test designers go to great efforts to examine local curricula in determining the subject matter they include, but this cannot completely solve the problem.
Consider the area of state history. The State of Colorado, for example, in their model history content standards calls for K-4 students to “chronologically [organize] significant events, groups, and people in the history of Colorado ” and to learn to create “a brief historical narrative that chronologically organizes people and events in the history of their family heritage, school, neighborhood, local community, or Colorado [Colorado Model Content Standards for History, 1995].” As adhering to these standards is a significant component of school assessment (and hence distribution of tax dollars), they form a significant part of state elementary school curricula. In fact, a course in Colorado history is required for college students who want to earn an elementary school teaching certificate. However, the chance of any questions on Colorado history appearing on an NAEP exam is just about zero. The subject is too parochial for a national test.
The same can be said to a different degree with regard to emphasis within curriculum. Students in the western United States are more likely to be trained on the history of the frontier than in colonial history. Likewise, students in Massachusetts undoubtedly get hit with Puritan history from an early age because those events happened in their backyard so to speak. The NAEP, on the other hand, is supposed to cover all geographic areas and time periods relatively evenly. The test does include questions on African American history, but schools that are predominantly African American will likely allocate more time to this branch of history than is allocated to it on the test. The NAEP attempts to reflect the diversity of history are not a bad thing, but no matter how hard they try the tests designers won’t reflect the curriculum of every school in the country. One size cannot fit all. If students aren’t taught a particular subject, it doesn’t mean they aren’t learning history.
Many of the multiple-choice questions on the NAEP are designed to get at whether students know particular facts. Knowing the author of the words, “We hold these truths to be self-evident: That all men are created equal,” is something that most Americans could agree should be taught before the Eighth grade, but a short answer question in the 2001 test for the same grade level was “Why was the invention of the steel plow important in United States history? [NAEP U.S. History Sample Questions,1994, #146 and 2001, #57]6 ].” If a middle school doesn’t cover this subject, are they raising bad Americans? This is not to suggest that the steel plow is unimportant, but there is a world of important ones out there for history and civics teachers to teach. Different school districts and different teachers will undoubtedly pick different facts.
But what about those facts that Americans can agree on? 71% of students answered the Thomas Jefferson question mentioned in the last paragraph correctly. That’s not cause for alarm. Another question from the 1994 test was:
What is the main reason the Pilgrims and Puritans came to America?
A) To practice their religion freely
B) To make more money and live a better life
C) To build a democratic government
D) To expand the lands controlled by the King of England
[1994, #129]
Knowing that this was a question designed for fourth graders, it is obvious that the right answer is A, but at least two other answers (B&C) are right to a degree. The key word in that question is “main” and that judgement is subjective. It should come as no surprise then that only 41% of students answered with the answer that the test designer’s wanted.
Ambiguities in open-ended questions are perhaps even more severe. Eighth Graders in 2001 were asked to “list two issues that were important to causing the Civil War.” After that, they have to explain the stand of “many Northerners” and “many Southerners” that turned the issue into a reason to go to war [2001, #53]. Only 9% of students got full credit for this one, even though the question is enormously broad. However, the fault here probably lies more with the design of the question than the students who answered it. Consider what are perhaps the two most widely cited causes of the Civil War, slavery and states rights. One is a northern reason, the other is a Southern reason. At this time, partisans of each side (and if you don’t believe there are partisans of each side still around, go see the web site of the Sons of Confederate Veterans) would deny the other reason’s validity. The answer key does not tell us if both are acceptable answers. Furthermore, why does it have to be two reasons? As James Brewer Stewart begins his essay on the causes of the Civil War in the Oxford Companion to United States History, “Modern historians agree that the problem of slavery was central in causing the American Civil War.” That is only cause he cites in his entire essay on this subject (Stewart, pp. 129-30). Any student who tried to duplicate this widely held position would fail to get full credit on this question.
Problems with curriculum match and ambiguity only distort the reliability of the NAEP as an indicator of student learning. The nature of the subject matter that the test does cover is a bigger threat to informed democracy because it reflects the conservative consensus championed by the test’s political backers. And because these views are cloaked under the auspices of a supposedly objective test, parents, students and others segments of the American public don’t realize how their understanding of history is being manipulated.
In preface to their handbook describing the framework for the 1994 and 2001 U.S. History exams, the National Assessment Governing Board writes:
As schools teach U.S. history, we hope they will foster a sense of excitement and pride, that students will gain a sense of who Americans are and what the idea of America is. Indeed, the nation was founded on a core of ideals – liberty, equality of opportunity, self-government under law, and respect for individual worth-that continues to draw millions to our shores.
These ideals were expressed at the beginning of the American Revolution, embodied in the nation’s founding documents, and restated by President Lincoln in his Gettysburg Address and more recently by Martin Luther King, Jr., in his speech at the Lincoln Memorial. The ideals serve as a force for unity and continue to animate reform. [National Assessment Governing Board, n.d., p. viii].
These words are an almost perfect embodiment of the Consensus School of American historiography [In fact, the subcommittee that developed the framework is called the NAEP U.S. History Consensus project.] ““Consensus,” writes historian Peter Novick, “became the keyword in [post-World War II] attempts to produce a new interpretive framework for American history, focusing attention on what had united Americans rather than what had divided them [Novick, 1988, p. 333].”
The consensus school of history is now, in the words of one prominent historian who lived through that era, “largely discredited [Higham, 1989, p. 255].” One reason for this academic consensus against the Consensus school is its status as a Cold War artifact. Daniel Boorstin is perhaps the best-known Consensus school historian because of his productivity and longevity. His 1953 testimony before the House Un-American Activities Committee explains how his perspective was shaped by events of that time. He stated that in his “attempt to discover and explain to students, in my teaching and in my writing, the unique virtues of American democracy” as a form of opposition to Communism [Boorstin quoted in Higham, 1989, p. 328]. The Cold War is over, yet the Department of Education is still designing tests with a Cold War mindset.
The other reason that Consensus history is no longer respected in academia is that it was rendered obsolete by the historiographic innovations of the 1960s. If the Consensus school still dominated the profession, African-American history, women’s history, labor history and scores of other sub-fields would not be taught. But to conservatives, this wouldn’t be such a bad thing. For example, Senator Lamar Alexander, Republican of Tennessee, blamed multiculturalism and diversity for America’s ignorance of history when he recently introduced legislation to improve the teaching of this subject: “During [the 1960s and 1970s], many of our national leaders contributed to this drift toward agnostic Americanism. . . . These leaders celebrated multiculturalism when there should have been emphasis on a common culture and learning English in unity [Alexander, 2003].”
The designers of the NAEP have not ignored the diversity of history on their two most recent U.S. History tests. Instead, they have incorporated new icons and institutions into the American consensus. Martin Luther King is the subject of three of the 244 sample NAEP questions available on its web site [2001, #14 1994, #123 &124]. Two refer specifically two his “I Have a Dream” speech and one refers to his use of nonviolence. Granted, these are the aspects of King’s life known to most Fourth Graders, but why don’t the test designers expect students to know more? Black students probably do. The answer is probably that King’s Poor People’s Crusade or his efforts on behalf of Memphis garbage men contradict the NAEP’s conception of a common national consensus.
Another example of this co-optation process is a question about trade unions on the 1994 test. It asks:
What is the purpose of labor unions?
A) To protect jobs and interests of workers
B) To help pioneers settle in the West
C) To help people move to other countries
D) To end slavery for African Americans
[1994, #95]
Aside from the fact that you don’t really need to anything about American history to answer this question, it neatly cuts out a huge segment of the American labor movement from the historical mainstream. Workers who joined unions for their own material benefit are exhibiting the consensus desire for upward mobility. Wobblies, communists and socialists are therefore not worth noticing. The same holds true for unions that engage in political activity. Eighth graders in 2001 had to answer this:
A major weapon unions have used in their struggles against factory owners is:
A) lockout
B) strike
C) voter registration drive
D) party convention
[2001, #55]
Their answer is B, even though the Congress of Industrial Organizations employed voter registration as a tactic often in the late-1930s and 1940s [See, for example, Zieger, 1995, p. 182]. And you can imagine how happy conservatives would be if unions stopped getting involved in grassroots politics today. This question de-legitimates an important form of liberal political activism.
The NAEP also encourages schools to teach patriotic myths. In 1994, fourth graders were asked, “When the Pilgrims celebrated the first Thanksgiving, what were they thankful for? Name three different things in your answer [1994, #115].” James W. Loewen, in his priceless study of school textbooks, Lies My Teacher Told Me, gets at many factual problems with the history that underlies this question. For example, he writes:
The Pilgrims did not introduce the tradition [of Thanksgiving]; Eastern Indians had observed autumnal harvest celebrations for centuries. Although George Washington did set aside days for national thanksgiving, our modern celebrations date back only to 1863. During the Civil War, when the Union needed all the patriotism that such an observance might muster, Abraham Lincoln proclaimed Thanksgiving a national holiday. The Pilgrims had nothing to do with it; not until the 1890s did they even get included in the tradition. For that matter, no one used the term Pilgrims until the 1870s [Loewen, 1995, p. 87].
Questions like these that center on myths rather than history are another sign that the actual historical knowledge of America’s students could be better than NAEP scores indicate.
Despite language in the descriptive material covering the test talking about the need for students to develop analytical skills, the decision of the test designers to focus on subjects that legitimate consensus caters to those with politically conservative view of history. Conservatives support the NAEP as a check to see if schools are presenting an uplifting, inspirational portrait of American history, not whether students are actually engaged in critical and analytical study. The NAEP does not test for knowledge of facts that do not fit into this conservative consensus or it manipulates the information so as to reflect a narrative of continual forward progress.
Twenty-seven states require high school students to take standardized history and/or social studies tests that share many of the same structural flaws as the NAEP [Education Week, 2003]. In some states, like New York, students have to pass these tests in order to graduate [Lynd, 2000]. In other states, like Texas, these test results are used to help determine how much money schools get from the state government [Benton, 2002].
Standardized history tests have become popular in recent years because of a huge change in American education in general. In exchange for accepting higher taxes, the business community has pressured school boards for educational accountability through standards-based testing [Pierce, 2003]. While the use of standardized tests may have helped improve student learning in subjects like reading and math (although this point is extremely controversial), in history this movement has dumbed-down the curriculum by expecting teachers to cover too much material in too little time. The more history teachers have to cover in class, the less time they have to consider their subjects critically. And as one educator explained in his analysis of the history and social science standards in Massachusetts, “Without [the opportunity for analysis, students] will be deprived of the kind of truly rigorous educational experience that not only makes the information that they learn meaningful, but which develops the intellectual know-how that is in fact the key to participating intelligently, justly and responsibly in civic life [Sills, n.d.].”
Instead of looking closely at standardized history tests in order to understand what they really tell us about the state of American civic life, conservatives use test results to forward their political agenda. In a Washington Post op-ed piece, the conservative historian Gertrude Himmelfarb uses standardized test scores to argue that, “A culture that values irony above all else, that is in the habit of traducing public figures . . . that cannot credit the very ideas of nobility or honor or heroism, or love of country, finds it all to easy to be irreverent, or, worse, cynical about the past, as about the present [Himmelfarb, 2003].” In a government controlled by an activist Republican President, this argument becomes an excuse too use history to discourage dissent.
In fact, the war in Iraq has offered the best indication yet of how conservatives only want to teach history that suits their agenda. On March 7, 2003, The New York Times ran a story entitled, “Schools Seek Right Balance As Students Join War Debate.” Chester E. Finn, Jr., a former assistant secretary of education in the Reagan Administration commented on these efforts by saying, “The purpose of schools is not to turn our 10-year-olds into policy wonks. Students should be learning the multiplication tables and the Declaration of Independence [Dillon, 2003].” So here, confronted with evidence that students are flexing their intellectual muscles considering the greatest civic crisis of the day, a conservative educational reformer wants to shut down the debate because a policy choice by a Republican President doesn’t need to be debated. In this instance, Finn wants to use history to prevent debate rather than provoke it.
The same is true of President Bush. To him, support for his war policy is the inevitable result of learning American history. As he stated in his September speech, “[Our children] are seeing Americans fight for our country, they also must know why this country is worth fighting for [Bush 2002].” By this logic, using history to decide that a war is a bad idea is not an acceptable outcome. Under these circumstances, the idea of informed democracy that presumably motivated Bush’s September 2002 speech is completely destroyed. And we are all the worse for it.
Alexander, Lamar. “Remarks of Senator Lamar Alexander on the Introduction of His Bill: The American History and Civics Education Act,” March 4, 2003.
Benton, Joshua. “Recent history often mystery to students.” Dallas Morning News, May 14, 2002. http://www.dallasnews.com/dmn/news/stories/051402dnmetabouthistory.52136.htm.
Bush, George W. “President Introduces History & Civics Initiatives,” September 17, 2002, http://www.whitehouse.gov/news/releases/2002/09/20020917-1.html.
Colorado Department of Education, Colorado Model Content Standards for History, September 14, 1995, http://www.cde.state.co.us/cdeassess/hist.htm.
Dillon, Sam. “Schools Seek Right Balance as Students Join War Debate.” New York Times, March 7, 2003, p. A1.
Education Week, Quality Counts 2003: The Teacher Gap. “Grade-by-Grade Testing Policies,” 2003. http://www.edweek.org/sreports/qc03/reports/testing-t1b.cfm.
Higham, John. History: Professional Scholarship in America, updated paperback edition. Baltimore: Johns Hopkins University Press, 1989.
Himmelfarb, Gertrude. “Proclaiming Our Principles.” Washington Post, February 16, 2003. http://www.washingtonpost.com/wp-dyn/articles/A18525-2003Feb16.html.
Loewen, James. Lies My Teacher Told Me. New York: The New Press, 1995.
Lynd, Christine. “The New Generation of Standardized Testing.” The Center for Education Reform, 2000. http://www.edreform.com/index.cfm?fuseAction=document&documentID=625.
NAEP U.S. History Consensus Project, “U.S. History Framework for the 1994 and 2001 National Assessment of Educational Progress, n.d., .pdf available at http://nces.ed.gov/nationsreportcard/ushistory/.
National Center for Educational Statistics, “National Assessment of Educational Progress Achievement Levels, 1992-1998 for U.S. History,” July 2001. Susan Cooper Loomis and Mary Lyn Bourique, Eds. .pdf format at http://nces.ed.gov/nationsreportcard.
National Center for Educational Statistics, The Nation’s Report Card: Parents Guide to the NAEP, n.d., .pdf format at http://nces.ed.gov/nationsreportcard.
National Center for Educational Statistics, Questions (Home), http://nces.ed.gov/nationsreportcard/ITMRLS/search.asp?picksubj=History.
National Center for Educational Statistics, U.S. History – The Nation’s Report Card (Home), http://nces.ed.gov/nationsreportcard/ushistory/.
Novick, Peter. That Noble Dream: The “Objectivity Question” and the American Historical Profession. New York: Cambridge University Press, 1988.
Pierce, Charles. “Testing Times.” The Boston Globe Magazine, March 2, 2003.
Sills, Jon. “Democracy, MCAS and the History and Social Science Framework.” N.d. http:///www.fairtest.org/care/Democracy,_MCAS_and_the_History.html.
Stewart, James Brewer, “Civil War: Causes.” The Oxford Companion to United States History, Paul Boyer, ed. New York: Oxford University Press, 2001, 129-30.
Vojeda, Barbara. “Judges Reject Census Sampling.” Washington Post, August 25, 1998. http://www.washingtonpost.com/wp-srv/politics/daily/aug98/census25.htm.
Wineburg, Sam. Historical Thinking and Other Unnatural Acts. Philadelphia: Temple University Press, 2001.
Zieger, Robert H. The CIO, 1935-1955. Chapel Hill: University of North Carolina Press, 1995.
1. My thanks to Tim McGettigan, Dave Mirabella from FairtTest.org and especially Radical Pedagogy's three anonymous readers.
2. The initiatives President Bush proposed on September 22, 2002 included grants for teachers to help them improve curricula and the posting of primary sources from the National Archives online so that educators everywhere can use them as teaching tools. But the problem with Bush's speech is not the solutions he proposed; it is his diagnosis of the problem, as the ensuing article suggests.
3. Those subject areas are reading, mathematics, science, writing, U.S. history, civics, geography, and the arts.
4. A civics test was first given in 1969 when the program began.
5. I'm relying on the U.S. Department of Education's description of the sample for these figures. See http://nces.ed.gov/nationsreportcard/pubs/guide/ques18.asp.
6. All questions in this essay are taken from the 244 U.S. History sample questions available on the NAEP U.S. History web site. I will footnote them with the year and the question number given on the web site as of March 24, 2003. Not all of the questions on the web site are as troublesome as the ones I quote here. However, because multiple-choice and fact-based short answer questions can only test for student knowledge of specific factual information, every question reflects the structural flaws inherent in all standardized history tests to some degree.
© Radical Pedagogy