Renowned scholar Les Perelman has dedicated his career to the support of powerful and authentic writing instruction. He has been an outspoken opponent of assessment practices he sees as counterproductive to developing strong writers. We wanted to learn more about the role he sees teachers playing in both advocacy for better assessments and the development of them.
- What role, if any, should teachers play in the scoring of assessments? Why?
Teachers should be involved in all phases of the assessment process to ensure that the assessment instruments test what should be taught, not what is easy to assess. However, they should not be the only people at the table. The design team should also include assessment professionals, representatives of school boards and education departments, parents, and, I would propose, one or two high-scoring recent graduates. [These former students] should be part of both the teams designing the general formats of assessments and each specific test.
The selection of training samples for each scoring session should be done primarily by teachers, with input by assessment professionals. The actual scoring sessions, however, should be run and staffed by teachers. With Internet technology, tables of teachers can be situated anywhere, trained on the same samples, and grade papers from almost anywhere. The system can be designed so that teachers will not be grading the papers of their own students. The composition of the teachers grading these tests should be as diverse as possible, and students should see pictures of these groups to diminish the effect of stereotype threat on students of color.
There are two very powerful arguments that can be used in proposing teacher grading of high-stakes tests. First, it would encourage teacher buy-in and eliminate the current almost adversarial situation felt by many teachers where their best teaching practices and their own expertise are at odds with testing companies and professionals who design and grade the tests but never enter classroom. Second, the monies spent on the grading session would serve a double purpose. Not only would they fund the scoring of essays, but they would fund extremely effective professional development.
During my whole career in writing program administration, I observed that if we didn’t have opportunities for teacher grading sessions, we would have to invent them. Getting teachers in a room and engaging them in conversations about [. . .] the key features in effective and intellectually adept student writing is one of the best venues for professional development. Moreover, it is an excellent method for engaging teachers in other fields who may be involved in writing-in-the-disciplines initiatives.
- What advice do you have for people who want to take action to improve the way we assess writing in this country but fear for their jobs if they do?
This is a very difficult question. People have families and obligations, and it is not my place to tell someone to risk their job. There are, however, two strategies I can suggest that may be useful in some situations.
First, use the strategies presented in Linda Adler-Kasner’s excellent book The Activist WPA for reframing the conversation. Accept the general goals presented to you, but then argue correctly that the current implementation actually subverts those goals. Propose intellectually and pedagogically honest strategies to achieve those goals that reinforce best teaching practices. Use the same vocabulary, such as “college readiness,” but define those terms in relation to effective teaching and the document jointly developed by NCTE, WPA, and NWP, Framework for Success in Postsecondary Writing. In some contexts these strategies will not work, but in others, they have a chance.
The second strategy is for teachers who are also parents. While it may be dangerous to oppose mindless testing where you work, as a parent and a taxpayer, you have a constitutionally protected right to do so in the district where you live. As a teacher, you can help lead the opposition by building alliances with other parents. Meanwhile, if the teachers who live in the school district where you work take similar actions, we are all helping each other, and most important of all, our children and students.
- In light of all you’ve learned in your research and through tools like the Babel Generator, where do you find hope in the future of assessment? What’s the bright spot in a landscape that looks pretty bleak?
Fortunately, I do not think the landscape is that bleak. What the BABEL Generator proved was how stupid these machines are. They do little more than count obvious and often trivial features. It took my team just a few weeks to build our first prototype, which we expected to fail with at least some machines. I still remember sitting in my office with my three students and trying out the alpha version of the BABEL Generator. We were amazed that it received top scores on all four machines we tested it on.
The most effective argument against automated essay scoring does not rely on abstract rhetorical theory. It is simply that these machines do not work; they do not do what they claim to be doing. Students can be easily taught to generate essays that receive high scores but that are atrocious pieces of writing.
These are arguments that everyone can grasp quickly, and the BABEL Generator is a tool that people can use to make these points in dramatic and unambiguous demonstrations.
Be the little boy who cries that emperor has no clothes!