Ed Reform 101: Testing

Blue Jersey’s Ed Reform 101

Part 1 – Standardized Testing Myths

Ed Reform 101In the world of the corporate reformer, standardized testing drives everything.

Judging teachers, principals, schools, and students; merit pay, tenure, and layoffs; allocating money; granting charters… it all starts with standardized testing. And it’s an article of faith among the corporate “reform” set that standardized tests are fair, accurate, inexpensive, and good for students.

The people who actually study this issue and work with children, however, know that nothing could be further from the truth.

There is a place for standardized testing in New Jersey, but it is inappropriate to use standardized tests in high-stakes decisions that affect teachers and students. We can’t measure a child’s learning or a teacher’s effectiveness when we put so much emphasis on secretive tests that are flawed in their construction, administration, and grading.

Yet almost every proposal put forward by the corporate reformers relies heavily on children filling in bubbles on a sheet of paper. So let’s start this series by taking apart the myths about standardized testing.

What you should know about standardized testing:

  • Standardized tests are typically imprecise, unreliable, and biased against the poor and minorities.
  • Too much emphasis on testing makes teachers focus only on what’s tested and encourages cheating.
  • Standardized tests are expensive, but they are graded by low-skilled, low-paid workers.
  • Student test scores are a poor way to evaluate teachers.

    and worst of all..

  • Too much standardized testing is bad for kids.

    (We encourage you to klick through all of these links: you’ll find some great information to help you arrive at your own conclusions.)

    Myth: Standardized tests are very accurate measures of student learning.

    The Truth: Standardized tests are incomplete and often inaccurate measures of learning.

    – Psychologists and education researchers agree: even the best standardized tests can only measure a small fraction of a child’s learning.

    Multiple-choice tests are especially “poor tools for measuring the ability to synthesize and evaluate information or apply knowledge to complex problems.”

    Myth: Standardized tests aren’t biased; they treat all students equally.

    The Truth: Standardized tests are often biased and unfair.

    – As FairTest, a national testing research and advocacy group, points out: “The damage created by high-stakes testing compounds rather than ameliorates the huge inequities caused by poverty and continuing racism.”

    A Stanford University study shows that standardized tests unfairly reinforce stereotypes minority and female students have of their intellectual ability.

    Myth: Standardized tests don’t change the way teachers teach.

    The Truth: Standardized tests lead to “drill-and-kill” teaching.

    – As the stakes get higher, teachers are prodded to “teach to the test.”

    – Teachers agree that standardized testing is taking time away from teaching important lessons in civics and other topics that are not – and could not be – covered by tests.

    Myth: Standardized tests are graded by well-trained professionals.

    The Truth: Tests are often graded by poorly trained and low-paid workers.

    – Standardized tests are usually written and graded by low-paid, low-skilled amateurs.

    Testing companies have advertised for item writers at the rate of $8 a question on Craig’s List.

    – Other reports confirm that test scorers usually make between $11 and $13 per hour.

    Myth: Standardized tests are inexpensive, they don’t drain dollars away from classrooms, and we have a good idea of what they cost.

    The truth: Standardized tests are expensive and NJ has never run a cost/benefit analysis to determine their worth.

    – The nationwide state-testing business is a billion dollar industry.

    – The Interim Report of the NJ Educator Effectiveness Task Force does not include a cost/benefit analysis of its implied plan to expand testing.

    – Only about 20% of the current teaching corps could be evaluated using current tests.

    Myth: Cheating on standardized tests is a small problem that can be contained with a few extra measures.

    The Truth: Cheating on standardized tests is running rampant, and even a huge investment of money into test security won’t stop it. Reports of cheating include New York City, Atlanta, Washington D.C. (while the district was run by the darling of corporate reform, Michelle Rhee), New Jersey, Texas, California and Pennsylvania.

    – It will undoubtedly cost much more than the current $18 million yearly contract to secure these tests, and local districts will face yet another unfunded mandate.

    Myth: Standardized tests are useful even for the youngest children.

    The Truth: Children under age eight should NOT take standardized tests.

    – Experts agree that standardized tests for children younger than eight are unreliable, unnecessary, and counterproductive (p.41).

    – The National Research Council cautions: “In general, large-scale assessments should not be used to make high-stakes decisions about students who are less than 8 years old or enrolled below grade 3.”

    Myth: Standardized tests are an excellent way to evaluate a teacher’s effectiveness.

    The Truth: Standardized tests are a terrible way to evaluate teachers. More on this later in this series.

    For more information on standardized tests, we recommend:

  • www.fairtest.org

    Next in our series: Teacher Quality

    Tuesday, 8/30/11: Standardized Testing

    Wednesday, 8/31/11: Teacher Quality

    Thursday, 9/1/11: Merit Pay, Seniority & Tenure

    Friday, 9/2/11: Teachers Unions

    Sunday, 9/4/11: Charter Schools & Vouchers

    Tuesday, 9/6/11: Recap

  • Comments (25)

    1. William Weber (WjcW)

      They never say why they are discriminatory.

      They make this completely circular argument… decades of research has shown low income and minority students score lower on standardized tests. But why?

      Why do they score lower? They never address it.

      They just say that because they score lower, the tests are discriminatory, based on the fact that those groups of students score lower.

      Does that make any sense?

      Or doesn’t that matter?

    2. the_promised_land

      NJ has had standardized testing forever. I remember taking CTBS standardized tests back when I was going to public school in the 80s.

      So how do they get to be the new thing and a big deal? And why do some people think that something that has been around so long is so critical to improving school quality?

    3. HurtPillow

      The old college educated people writing the tests just do not KNOW the diversity in the state.  The first grade test had the word ‘cupboard’ on it and that is not a word in their vocabulary.  This begs the question, if the tests were skewed with the vocabulary and experiences of urban youth, how would white suburbia fare?  This is one limited example of a systemic issue in the testing genre…  Yes, tests are now a literacy genre according to the Literacy Collaborative.  How sad is that?

    4. Mazel tov to Blue Jersey for this series and to Jersey Jazzman for an amazing kickoff.  

      As some of you know, I personally (speaking only for myself here and not any organization) have mixed feelings on the topics that the series will cover.  For instance, I would support the increased use of standardized testing IF it could be made fair for disadvantaged students and for all the other reasons Jersey Jazzman has presented.  I recognize, too, that may not be possible.  

      Regarding topics later in the week, I do have problems with lifetime tenure in any profession that serves the public.  I believe there is more room for accountability.

      BUT:  I also recognize a great series and a great article.  I found Jersey Jazzman’s piece here to be thoughtful and so well laid out – concise and direct points that are reader-friendly and and very thought-provoking.  Certainly made me think – so thank you, thank you, thank you.  

      With appreciation, Steven

    5. Jersey Jazzman (Post author)

      “In more than a dozen experiments over the past four years, Steele and his colleagues were able to depress the performance of high-achieving African American men and women of all races by subtly implying that well-known stereotypes about those groups intellectual abilities might apply to the test they were about to take. The cues were often indirect. Students were told that the test they were about to take can measure ability or they were asked to mark down their races before the test began. In control groups where similar students were given no reason to suspect that the demeaning stereotypes would apply to their performance, African Americans performed as well as whites on very challenging tests.”

      I mean, that’s awfully strong. How much more do you want?

    6. Jersey Jazzman (Post author)

      Are you prepared to say that the problem is that minority students, low income students, and girls are innately inferior, and that the tests are merely showing that?

      Or does it make more sense that the tests are biased against these students?

    7. SmartyJones

      My non-scientific theory posits that in generalized, not subject specific, achievement tests, the preparers draw from what they consider “general knowledge.”  Unfortunately, many students are not exposed to what the preparers think is common wisdom.

      Here’s a perverse/reverse, albeit simplistic example.  My nephew, who comes from a decidedly upper middle class household, as a youngster,  got a question wrong on one of these tests because he didn’t know what an iron was.  It’s not that my sister had “help”; she just never ironed.  She probably had an iron – it just never came out of the closet.

      In high school, nephew was unhappy and his grades reflected it.  He’s bright, an avid reader and talented in math.  It’s just what he wanted to know had nothing to do with the school curriculum. Maybe he had more generalized knowledge than others who aced their courses, because he got a near perfect score on the SAT’s.

      As for me – I will never get a correct answer on the ubiquitous question of “Train A leaves point X, traveling at 80 MPH and Train B leaves point Y, traveling at 60 MPH.  I don’t remember the rest of the question, or the answer because I hate it so much.

    8. Jersey Jazzman (Post author)

      It’s not that standardized tests are “bad” per se; it’s that you have to understand their limitations to use them correctly.

      They can be very helpful in “low-stakes” decision making: curricular choices, program assessment, etc. They can absolutely provide feedback for a teacher or principal into practices, provided they are taken into context, which is highly individualized. They can provide some additional data about a child when making decisions about program placement – again, when taken in context.

      Where they fail miserably is in “high-stakes” decisions: teacher pay, school funding allocation, student retention. They simply aren’t up to the job.

      That’s what’s changed from when you and I took them back in the day. No Child Left Behind has completely distorted policy makers’ views of what these tests can and should do.

      Put another way: the tests haven’t necessarily changed; what’s changed are the politicians’ use of them.

      One more thing: there’s a lot of evidence that the tests themselves are too often poorly constructed and vetted. Look at what just happened in New York, where the state admitted the tests were getting easier each year:


      In NJ, a teacher isn’t even allowed to see the test – how is she supposed to adjust her teaching when she can’t even judge the instrument that is judging her?

      Again, it’s not that we shouldn’t have tests – it’s that we shouldn’t use them in inappropriate ways.

    9. Jersey Jazzman (Post author)

      This is a little lengthy, but it’s a great post by Todd Farley about just this:


      What constitutes a “favorite food” for a 4th Grader determines whether he or she passes the test. And in the corporate reform world, that would redound to the teacher’s pay. Tons of potential bias here related to culture.


    10. William Weber (WjcW)

      but what about math.

      Don’t the results follow in math in which questions are more than likely ‘culturally neutral’?

    11. William Weber (WjcW)

      but it’s hard for be to believe that if I tell someone the test is too hard for them, they score worse… which is essentially what that study says.

      Were they able to prove it the other way… implying that tests would be harder for whites? Or easier for minorities?

      The study’s results are the results, I get that, I just think that we are missing something.

    12. William Weber (WjcW)

      That’s not my thought at all.

      Just like I don’t think Chinese students are innately superior.

      By and large I think you will find Chinese students score better than whites on tests, would you say the tests are biased against whites?

    13. Jersey Jazzman (Post author)

      … is hardly culturally neutral.


      According to Kessel and Linn, there is no question that the test is at fault. Summarizing more than a dozen studies of large student groups and specific institutions such as MIT, Rutgers and Princeton, they conclude that young women typically earn the same or higher grades as their male counterparts in math and other college courses despite having SAT-Math scores 30-50 points lower.”  

    14. Jersey Jazzman (Post author)

      … the tests are biased.


      Again, your construction here dictates that either certain groups are inherently intellectually superior, or the test is biased.

      Pick one. You can’t have it both ways.

    15. William Weber (WjcW)

      You believe the tests are biased against whites? Or Chinese are inherently superior?

      Isn’t it possible that Chinese culture puts a higher value on education/work ethic and that’s why they score better?

    16. Jersey Jazzman (Post author)

      You must not be reading the links I’ve posted.

      Very often, the tests are biased. They reinforce cultural stereotypes in a way researchers have replicated.

      So, yes, the tests may very well be biased toward Asian-Americans, as there is a cultural stereotype about their intellectual capabilities. Steele’s research shows those stereotypes are reinforced in tests. Showing one group has a bias toward the test in no way contradicts a conclusion that a bias away from the test shows the test is biased.

      Further: you have evaded my question. Do you believe there is an inherent inferiority in African-Americans, women, or other groups that is responsible for differing results on standardized tests? I do not.

      I have to say, William, that you remind me very much of the governor and others who do not have much time for rigorous research. I am putting high-quality studies in front of you; you dismiss them with your hunches. You say “we are missing something.”

      That is a very low bar for this discussion.

    17. KendalJames

      It’s quite tiresome to read your comments on this thread when every single question or point you raise is already addressed in the links provided. Above, you essentially acquiesce by stating that, when presented with the evidence you request, you don’t like the findings and so therefore something must have been overlooked. PLEASE approach this subject with intellectual honesty and an open mind, which in my humble opinion starts with actually reading the materials before questioning them.

    18. Jersey Jazzman (Post author)

      I appreciate that.

      We’ll get to tenure later this week. Without giving too much away: everyone agrees that tenure needs to be changed, including the NJEA. How and why it needs to be changed, however, is a topic that has not yet had much serious debate.

      Stay tuned, and please post your thoughts Thursday after you read the post.

    19. KendalJames

      This language perturbs me. What is it about workplace protections and the right to due process that should be earned? I worry that the myth of tenure as a “job for life” has become the dominant paradigm despite it not being that, at all. Changes to the process to make it faster, cheaper, more efficient, even more fair and judicious? Absolutely. Making an individual’s right to workplace protections based on variables that individual cannot control? No.

      Unfortunately, high-stakes testing as a measure of teacher effectiveness has never – not a single time – been proven to be a useful or accurate metric. I’m sure JJ will let us know the deal.  

    20. the_promised_land

      I look forward to seeing this post and learning more.

    21. Jersey Jazzman (Post author)

      … Asian and Asian-American. Two entirely different groups, obviously.

      You are, however, absolutely right about “drill-and-kill” and the Chinese education system. The Chinese education leaders themselves are worried about it:


      “The failings of a rote-memorization system are well-known: lack of social and practical skills, absence of self-discipline and imagination, loss of curiosity and passion for learning. Chinese students burn themselves out testing into university, where many of them spend their time playing World of Warcraft.

      Both multinationals and Chinese companies have the same complaints about China’s university graduates: They cannot work independently, lack the social skills to work in a team and are too arrogant to learn new skills. In 2005, the consulting firm McKinsey released a report saying that China’s current education system will hinder its economic development.”

      So we’re running the risk of becoming the very thing the Chinese themselves are trying not to be.

    22. the_promised_land

      … I’m sure that would happen in many places. I’m interested to read later in the week how any tenure reform would deal with that including what the NJEA is proposing. Because I can’t personally support any reform that doesn’t directly deal with this problem in a serious way.

    23. Jersey Jazzman (Post author)

      It’s a serious problem that I cover on Thursday.

      I wanted to deal with testing first, however, because it’s the center of all of the “corporate” reforms. The tenure changes the Christie admin and his Task Force propose rely heavily on standardized, high-stakes testing.

      But if there are serious flaws with the tests, why argue to use them in tenure cases?

      This also feeds into teacher quality, which we will deal with tomorrow. Stay tuned.

    24. William Weber (WjcW)

      the whole study essentially says people do worse on tests (but doesn’t give any quantitative measure of how much worse) because they believe they should do worse and/or put too much pressure on themselves to overcome the belief that they should perform worse.

      Wouldn’t that issue would manifest itself in any objective measure?

    25. William Weber (WjcW)

    Leave a Comment

    Your email address will not be published. Required fields are marked *