Monday, January 02, 2012

Get Those Numbers Out of Here or I will Call the Authorities

The other day I spoke to a colleague about the fact that my school has two grading curves depending on the size of the class. I wondered allowed if we could study whether the 2 curves influenced course selection, class rank, and GPA. The response was not argumentative at all but more or less "I'm not that interested in numbers."

That is, in fact, a common theme in legal education and represents how far legal educators are from the applying various principles and measures that would follow from good educational policy.

The example I have given is directly related to the issue of student choices. Do we want students to select courses that will best prepare them for the practice of law (or tending bar as the case increasingly seems to be) or do we want them to be tempted to game the system.

The two curve example reminds me of one of the worse testing strategies around: Please select and answer any 3 of the following 5 questions. Yes, it's a policy that says to pick the test you would like to take. The student who would get an B on three quesitons and a C on the other two gets the same grade as the student who would get a B on all 5. Like the different curves, the testing method itself intrudes on the process in a way that is disconnected from the goal. But there too we get into numbers.

Something that also falls in the area where law professors do not venture is the reliability and validity of exams. Reliability is really a question of consistency. For example, you turn the hot water tap on half way for your bath and the water is always 90 degrees. You can count on it. But, you also say 90 degrees is just right for making your aching muscles feel better. That is a question of validity.

Suppose every time you write and give an exam, there is a nice bell shaped curve. You might say your testing is reliable: every time you give an exam X happens. But, do you know anything about the connection between what you hope to be testing for and the outcome? This question of validity is a different matter. I am far from an expert but, let's say you give machine graded multiple choice exams. How do you know the questions are valid measures of what you want to measure. There could, after all, be 5 reasons to miss a question or get it right and only some are related to what you are testing for. I would guess that any multiple choice question that does not require a student to explain his or her answer would have to undergo testing itself and perhaps trial runs and debriefings of the students so see what they understood the question to be asking and how the different choices could be interpreted.

And then there is the matter of student evaluations. What do they tell us? I'd say they are very reliable and valid indicators of what the students wrote down on their forms. Other than that, I do not know. The problem is that no one else does. Wouldn't it be nice to know what the evaluations mean as far as actual student learning? I've seen studies that indicate no correlation between evaluations and learning and even some that indicate a negative correlation.

But then again, to actually attempt to determine what the numbers mean would mean dealing with numbers.

Finally suppose you offer three or four credit course you would like to teach in 2 days. There are all kinds of studies on the impact of different class lengths on concentration and learning. I wonder if any law professors have looked at these "numbers."

Sometimes it seems clear to me that numbers are discounted principally because they may tell us something we do not want to hear. The main thing we do not want to hear is anything that casts a shadow over whether we should get our way.


Stephanie said...

From the POV of a [former] student, I can tell you that the curve/grading definitely affects the classes I would take. Smaller classes = less chance of a C. Also, pass fail classes, or classes sans finals are also highly coveted. Of course, there are exceptions. I took skills classes for the skills and I took classes that interested me regardless of difficulty or size. However, I think the most important factor in what classes I chose was professor reputation.

Anonymous said...

Increasingly students are discovering that hiring firms actually do look at the assortment of classes that students take. Loading up on lots of cushy impractical courses is a bad idea. This comes as a shock to many students.Firms realize that they must play a role in advancing legal skills. They do not like encountering potential employees who have them starting the process and base zero.

Anonymous said...

The curve does matter! Although law students want to be intellectually stimulated they also don't want to receive a low grade for a class they worked very hard in. So students learn to game the system. This means selecting subjects, teachers, and curves that are in their best interest. After all, having a higher GPA helps one narrow the chances of getting a much needed job.

Some students that even have high GPAs take classes with the higher curve so that they can ensure their higher ranking in the class overall. If you look at your institution, I bet that you'll see that the Law Review kids are the ones dominating the higher curved classes.

Also, I'm sure a firm will overlook class selection if someone made Order of the Coif.