The painful truth about our exams

Don’t ask a South African ­matriculant to do a sum, balance an account, give directions or name a plant – but they are pretty good at giving you some lip.

The truth is our Grade 12 graduates are great at languages but startlingly poor at other technical and more ­academic subjects. No newsflash there. But what we generally do not know is the detail of how great or how poor they are. And that was the story that my colleagues and I in Media24’s ­investigations unit were told in some detail recently when we looked under the hood of the matric results.

The story produced a rather sharp riposte from Umalusi, the exams quality assurance body, who didn’t like our data and conclusions.

The matric story lends itself quite naturally to a growing field called data journalism – a field in which I have an intense personal interest. So my team and I thought it would be useful to look at how last year’s matric class performed in each subject for the actual exams, and to compare multi-year ­data to get a sense of the reality behind the pass rate as well as the trends.

The overall matric pass is based mainly on the year-end exams but also includes a portion – about 25% – which comes from the work that a ­pupil did during the year, the assessment standard of which can differ wildly. All this has a significant effect on the matric pass “bottom line”.

How a pupil performed in each exam paper is a clear indication of ability in that subject. After all, that’s why we do exams. To understand this, we needed to collate data from several years of performance in exams. In many countries that would be a breeze to do, but not in South Africa.

Getting access to official data like this is rarely possible without screaming tantrums or recourse to the law.
Last year media organisations,­ ­including Media24 Investigations, filed freedom of information applications to Umalusi to get the raw marks for the 2010 exams. The purpose of this was to see what the results were like prior to the exams body’s standardising ­adjustments came into effect.

In that trove of documents was a breakdown of raw and adjusted marks for each matric exam going back to 2008 when the first group of matriculants in the new curriculum graduated. And that gave us part of the data we were looking for.

We then asked Umalusi for the same information for last year’s exams. ­And after much hemming and hawing, they agreed to hand it over.
The results were shocking but formed the basis of our story. We were able to enrich the data gleaned from 2010 with other information from Umalusi’s yearly “technical report”.

Our article led to Umalusi’s rather sniffy response that there was something wrong with our story.
It was based on Umalusi’s own data, even if we had to tediously extract some of it from a morass of information. Numbers, of course, can tell you any story you want if you massage them enough. But at times, they also tell a simple – and even painful – truth.

» Trench is the editor of Media24 Investigations

» Tell us what you think: Leave your comment below or comment on our Facebook page or on Twitter @City_Press
» Did you know? City Press has an iPad app. Find us in the app store

- City Press