Education

The Precision of Exams

Back when I was at IIT Bombay, I often heard the refrain that the JEE was much tougher than the SAT, and this fact was offered as evidence of the high standard of the IITs and their students.

I do not want to discuss the standards of IITs here, but I do want to argue against “toughness” of exams as a measure of how good they are. An exam is a measuring instrument. A measuring instrument should be graded against how well it measures what it is expected to measure, and whether it delivers the precision that is required of it. It is obviously a bad idea to use a weighing pan to measure the width of a road, but it is also a bad idea to use vernier callipers for the purpose, or expect the width of a road to be accurate to the tenth of a millimetre.

Exams too should be evaluated against similar criteria. I can think of three types of exams, each type with a distinct purpose for conducting them, and I am sure there are more that I have not thought of. I’ll name them types 1, 2 and 3.

Type 1, exemplified by the JEE or CAT, are exams whose purpose is to select the very best of those who apply, and figure out fine gradations among them. The JEE assigns ranks to around 2% of those who take it. It has to reliably distinguish between the person who obtained the 10th rank and the 100th rank. It achieves this purpose by being an extremely tough exam. But the price it pays for being a difficult exam is that almost everyone below the top 2% scores close to zero, and the exam therefore fails to distinguish someone who performed at the 95th percentile from one who performed at the 85th.

That is where Type 2 exams come in. The SAT and GRE exams are of this type. Our board exams should belong here too, but I’ll have more to say about them later. The objective of these exams is to grade along a curve the entire population that takes them. Type 2 exams cannot be as tough as Type 1 exams are. They must have a mixture of questions of varying levels of difficulty. They can reliably distinguish between the 90th and the 60th percentile, but shouldn’t be used to decipher fine distinctions between the test-takers at percentile 99.2 and 99.5. The margin of error is too large to make such distinctions.

When elite American colleges use the SAT or the GRE in their admission decisions, they do not use the exam score as the only criterion. They use the score as a cut-off and then make the final decision based on other parameters, or use it as one criterion among many. That is the correct thing to do. The scores simply do not provide enough precision to be used on their own.
I often hear about proposals to expand the scope of exams like the JEE to turn them into exams for entrance into more engineering colleges than just the IITs. When you do that, you are imbuing more Type 2-ness to what was formerly a Type 1 exam. That is not a problem if you do it consciously and carefully. You also need to be aware that when you do that, the JEE will be less able to satisfy Type 1 requirements. It will not be possible to make fine-grained distinctions between top people using JEE results. I am concerned that our policymakers are too casual about these things and do not consider these things when making decisions.

Type 3 exams are the easiest of the three. They are like the assessments I have to pass to prove that I have paid attention to the mandatory web based trainings I have to take every year at my workplace. I work for an American bank. As part of a compliance requirement, all employees undergo trainings related to Know-Your-Customer (KYC) and Anti-money Laundering (AML). The tests I have to take at the end are easy. The passing score is 80%. The objective of the tests is to validate that everyone at the bank has the baseline knowledge of the contents of the courses. It is not to select the best among them. Getting 100% of the answers right won’t get you an automatic job offer from the compliance department. The objective isn’t even to grade everyone on a curve. Everyone eventually scores between 80 and 100 percent in the test, and there’s really no material difference between the capabilities of someone who scored 80 and somebody who scored 100.

I hope I’ve made the case for why the toughness of an exam isn’t a good measure of its quality and why we should take into consideration what the exam measures to determine how well it does it. How do our board exams measure up?

Board exams should be Type 2 exams. Their purpose is to grade everyone who takes them on a curve. A student who has achieved what we consider the lowest acceptable level of competence in the subject must get passing marks. The best students should get the highest marks. Between the highest and the passing scores, there should be a wide spread of scores that enable us to reliably distinguish test takers’ performances from one another.

Unfortunately, that is not how things are. Our board exams are too easy. I would estimate that the minimally competent student should easily score 75% in them. The boards set the bar too low because most schools have abysmal standards of education. If our board exams were calibrated to a reasonable level of difficulty, failure rates would be too high, triggering a political backlash. Once in a while, we see reports in newspapers that a particular CBSE exam had too many questions that were “out of syllabus”. I suspect that most of the time, “out of syllabus” just means that it didn’t conform to the pattern that students expected it to follow.

Why do these exams follow “patterns”? Apart from the political reason already referred to, there’s the expense of calibration. There aren’t enough qualified teachers to correct these papers. Preparing a key that will enforce consistency in marking is tougher for tough papers. It is far easier to have a limited set of questions with model answers provided, so that examiners can correct via pattern recognition.

The upshot of all this is that these exams, that should be type 2, are closer to type 3. Anyone with a baseline level of knowledge should score very high, and there is really not much difference between someone who scored 85% and someone who scored 95%.
In the hype that is created around these exam results and the impact that they have on the students’ prospects, they are type 1. Ranks are assigned over tenths of percentage points, and admissions to courses in premier educational institutions are won or lost over them.

So how do you “crack” an exam with predictable questions, and one that is corrected using pattern recognition? You mug up the model answers. You go to coaching classes whose goal is not to supplement the knowledge you’ve gained at school, but to replace what schools are supposed to teach you with a database of questions and answers in your mind.
I’ve seen people defending rote learning as one of the valid forms of gaining knowledge. The point though is that while there may very well be a case for learning some facts or multiplication tables or formulas by rote, learning model questions and answers by rote is not something anyone can reasonably defend.

These exams aren’t the primary cause for what’s wrong with our educational system, but fixing them is the point at which you should start, because if you fix the exams, the incentive to mug up things will reduce, and things will start changing down the line.

What the Government Says and What it Means

Ritwik says:

Your argument is essentially that regulations will ensure that only those who see a profit motive in education and are able to lobby with regulators will survive and the true educationists/ philanthropists will move out due to overburdening and ever increasing regulations. Isn’t this in contradiction to the usual lament that one of reasons why education lags in India is the fact that one can’t open schools for the profit motive? In such a case, a law that disallows the provisioning of education for the profit motive should keep these people out, right?

Ritwik is obviously not married and has never attended the Art of Living course.  If he had done either, he would have learnt that the words of your wife or of Sri Sri Ravi Shankar are never to be understood at the superficial level. There are always deeper levels of meaning to it. So it is with the words of the government.

Continue reading →

How Do You Find Good Schools?

In a curious response to my post, Ravikiran Rao points to an earlier post that draws an analogy between blogs and educational institutions. In particular, it should be easy for people to start educational institutions — just as it is easy for people to start blogs. You have heard about the death of set-up costs, haven’t you?

Students will be able to discover for themselves where the good institutions are, and they will flock to them — just like readers discover good blogs now. Death of transaction costs, too!

Ravikiran Rao seems very confident that he has found a clever solution to our higher ed problems.

Just imagine the possibilities of this wonderfully costless world: our students will be able to sample a whole bunch of colleges / courses for two minutes each. Or, they’ll just need to discover one or two good colleges, which will point them to many other similarly good colleges. (nanopolitan)

I did not intend to claim that I knew the solution to all the problems of higher education. I was merely pointing out that regulation is part of the problem, not part of the solution.  Regulations impose barriers to entry and they tend to penalize the honest the most. Being snarky does not free you from addressing the point.

His second point is of course, quite valid. If it weren’t for the government certifying the quality of schools, the only way to find out their quality is by enrolling in them and studying in those schools for two minutes. It is because of the regime of school inspection that has freed us from the burden of selecting schools for our children. It is a proven fact that middle-class parents who have to choose schools do so by draw of lots. It is also a fact that there is a social taboo against speaking about how good or bad your school is, which makes it impossible for parents to gain any information about the quality of schools.

Private Players in Higher Education Are Corrupt! How Horrifying!

Our current problem is not that there are no private players, but that they have among them too many crooks, politicians and thugs whose primary motive is demonstrably something other than education. So the real issue is this: What changes do we need in our regulatory structure so as to attract the ‘right’ sort of private players — philanthropists — to enter the education sector in larger numbers? How do we keep the undesirable kind from poisoning the pool? (nanopolitan)

Bzzt. Already answered. Next question please.

In Which I Avoid a Trap Set By Nilu

Nilu says that I do not address a certain argument in favour of government schools. The argument has something to do with poor people having the vote. If that is supposed to mean that the poor can vote themselves better schools, Nilu should know that it is nonsense. One vote every five years is simply inadequate as an attention-getting tactic, when citizens have a hundred issues on which to draw their rulers’ attention. Presumably Nilu does know that, so he modifies the argument with something else that is still nonsensical.

Continue reading →

The Real Lesson of Kendriya Vidyalayas

A couple of years back, at a blog meet, I was having a discussion with Anand, who used to blog at locana. He was trying to defend government schools. His defence went: “Not all government schools are bad. I went to one myself. Ok, it was a government school at a campus that was filled with professors, but still…”

I sputtered a bit but never got a chance to complete my response in the din of the meet. This post at Nanopolitan reminded me of that conversation. This is as good a time as any for a response, I suppose.

People argue that private schools will serve only the rich and never provide the same quality to the poor. When faced with evidence that government schools also provide good quality only to the rich and neglect to serve the poor, their views undergo a fascinating inversion. The success of private schools for the rich is evidence that they will serve only the rich. The success of government schools for the rich is evidence that hope is on the way for the poor.  

Continue reading →