Get Those Numbers Out of Here or I will Call the Authorities

The other day I spoke to a colleague about the fact that my school has two grading curves depending on the size of the class. I wondered allowed if we could study whether the 2 curves influenced course selection, class rank, and GPA. The response was not argumentative at all but more or less "I'm not that interested in numbers."

That is, in fact, a common theme in legal education and represents how far legal educators are from the applying various principles and measures that would follow from good educational policy.

The example I have given is directly related to the issue of student choices. Do we want students to select courses that will best prepare them for the practice of law (or tending bar as the case increasingly seems to be) or do we want them to be tempted to game the system.

The two curve example reminds me of one of the worse testing strategies around: Please select and answer any 3 of the following 5 questions. Yes, it's a policy that says to pick the test you would like to take. The student who would get an B on three quesitons and a C on the other two gets the same grade as the student who would get a B on all 5. Like the different curves, the testing method itself intrudes on the process in a way that is disconnected from the goal. But there too we get into numbers.

Something that also falls in the area where law professors do not venture is the reliability and validity of exams. Reliability is really a question of consistency. For example, you turn the hot water tap on half way for your bath and the water is always 90 degrees. You can count on it. But, you also say 90 degrees is just right for making your aching muscles feel better. That is a question of validity.

Suppose every time you write and give an exam, there is a nice bell shaped curve. You might say your testing is reliable: every time you give an exam X happens. But, do you know anything about the connection between what you hope to be testing for and the outcome? This question of validity is a different matter. I am far from an expert but, let's say you give machine graded multiple choice exams. How do you know the questions are valid measures of what you want to measure. There could, after all, be 5 reasons to miss a question or get it right and only some are related to what you are testing for. I would guess that any multiple choice question that does not require a student to explain his or her answer would have to undergo testing itself and perhaps trial runs and debriefings of the students so see what they understood the question to be asking and how the different choices could be interpreted.

And then there is the matter of student evaluations. What do they tell us? I'd say they are very reliable and valid indicators of what the students wrote down on their forms. Other than that, I do not know. The problem is that no one else does. Wouldn't it be nice to know what the evaluations mean as far as actual student learning? I've seen studies that indicate no correlation between evaluations and learning and even some that indicate a negative correlation.

But then again, to actually attempt to determine what the numbers mean would mean dealing with numbers.

Finally suppose you offer three or four credit course you would like to teach in 2 days. There are all kinds of studies on the impact of different class lengths on concentration and learning. I wonder if any law professors have looked at these "numbers."

Sometimes it seems clear to me that numbers are discounted principally because they may tell us something we do not want to hear. The main thing we do not want to hear is anything that casts a shadow over whether we should get our way.