Assessing the Ineffable

As we prepare for the SACS team’s visit next fall, numerous committees on campus keep fretting the issue of assessment. In order for our teaching to be accountable, it must be assessable, which means that we must define what we expect our students to be learning and devise measuring instruments that will determine whether they are actually learning those things—and then use the results of our assessments to improve our programs.

I am in complete sympathy with this goal. Unlike my colleagues who find the kind of accountability these assessment demands are looking for invasive, an assault on what they think of as their "academic freedom," I agree that we need to think about what we do and why, and find ways of determining how successful they are. I am strongly opposed to the notion that teaching and learning are fundamentally mysterious processes that don't bear close scrutiny: that you just go into class and "teach"--i.e., behave in a way that seems familiar from how you were once taught, decades ago--and hope for the best.

The problem I have with the assessment demands that we face, though, is epistemological. What epistemic relationship can be established between learning and assessment? Or, more specifically, even if we accept the need to define learning outcomes as specifically as possible, how can a quantitative assessment--which is what our administration has systematically demanded from us--tell us anything worthwhile about the extent to which those outcomes are being achieved? And even if we manage to set up an assessment plan that draws on both quantitative and qualitative measurements, is it ever possible to determine whether students are learning what we want them to learn?

For example, our QEP Design Task Force II found a list of student writing learning outcomes somewhere and asked me to tweak them a little. The first one is "students will demonstrate writing as a process that requires brainstorming, drafting, revising, editing, and proofreading." I hate the collocation "demonstrate writing," want to change that to "experience writing," but of course "experience" is harder to quantify than "demonstrate," so if I change that verb, most likely the task force will change it back. The empirical bias of assessment experts forces us to think in terms of externally verifiable (“demonstrable”) products rather than hard-to-verify inward experiences; and then we try to force an inward/experiential thing like "writing as a process" into the outward/verifiable mode, and end up with monstrous phrases like "demonstrate writing as a process."

Of course, maybe what I object to in that phrasing is just the unnecessary wordiness of "demonstrate writing." If we try to figure out what that really means, doesn’t it boil down in the end to just plain "writing"? The only way to "demonstrate writing" is to write, right? So maybe the desired learning outcome would be to "write as a process"? But then writing is always a process; the idea here is to make students aware of the writing process, and to get them to draw effectively on that awareness as they write their papers for class. So it's not just that we want them to "write as a process"; it's that we want them to write with an awareness of the ways in which the writing process requires them to pay attention to certain things as they proceed, and respond in certain ways to what that process draws their attention to. And that suggests that an "outward" and perhaps even verifiable formulation of "demonstrate writing as a process" would be "demonstrate awareness of the writing process," or perhaps, more fully, "demonstrate an ability to write with an awareness of the complexities involved at every stage of the writing process, including brainstorming, drafting, revising, editing, and proofreading."

But is that true? Is that really the desired student learning outcome? Does this mean that a student who writes brilliantly but intuitively, without that awareness, will fail to reach the desired learning outcome? What we want, ultimately, is for our students to write well; getting them to pay attention to the writing process, to undergo the various stages as consciously and conscientiously as possible, is a means to that end, a strategy that five decades of writing-process theory has identified as the best means to that end. The brilliant but intuitive writer may not need that means—may not need to unpack the process consciously—but in the end s/he too will experience writing as a process. But how do we assess that?