NCEA maths exam mistake: Expert panel surprised that no one picked up error

A mistake in a crucial maths exam that upset even excellent students last year was made in the first draft of the exam paper - and nobody picked it up.

An expert review panel says it is "surprised" that no one spotted the error even though "all the steps in the examination development process were followed".

The level 2 maths exam for the National Certificate of Educational Achievement (NCEA) was so tough that two Southland students set up a Facebook event to "get all Level 2 maths students to storm NZQA headquarters" - the head office of the NZ Qualifications Authority, which set the exam.

A student who was aiming for excellence said "everyone was just shocked" by numerous questions in the exam that they had not anticipated.

A student at another school said the majority of the students were unable to complete the paper - some were in tears and left after 45 minutes because they could not answer the questions.

The expert panel did not review the whole paper, but found that a mistake in question 2(d) of the algebra section of the exam meant that the answers from solving a quadratic equation required to answer the question were inconsistent with the diagram provided.

The faulty question 2(d) in the algebra section of the level 2 maths paper. Photo / Supplied

"Question 2(d) was a straightforward quadratic equation problem, in the context of areas of rectangles, which could be solved in several ways," the panel said.

"Not all of the methods would have caused candidates to refer back to the diagram. The error could only have been detected if a candidate re-checked their calculations by substituting the values for 'x' and 'y' back into the diagram."

The panel found that the error in the diagram "was created by the materials developer at the very first iteration of the paper's development".

"At no stage throughout the subsequent development and checking of the paper was the error identified by the examiner or any other of the examination development personnel.

"This surprised the subject-matter experts on the review panel who considered re-checking of calculations by substituting the values back into the original question/diagram to be normal practice in mathematics."

The question was intended to allow students to demonstrate "excellence" - in other words, it was meant to be harder than other questions so only excellent students could answer it.

But an independent expert who critiqued the exam paper before it was finalised pointed out that it could be answered using "a simpler method than the examiner anticipated".

Two other independent checkers also "provided alternative and easier methods to the question".

"However, the examiner did not amend the question."

"The panel notes that all the steps in the examination development process were followed and considers that there are an adequate number of required checks in the system," the panel said.

"There was only one issue raised during the development of the algebra paper – that of whether Question 2(d) was at Excellence level. The error remained undetected.

"Therefore, while the process was followed, the panel believes further steps could be taken to improve the process and thereby avoid a similar error occurring in future. In particular, there could be ways to ensure the materials critiquer's working is visible to the materials developer and/or the examiner."

The panel also recommended adding specific instructions in the NZQA examiners' guide to check all diagrams and ensure that any "numerical values expected in the solution are consistent with each other and with the diagram".

Despite the error, NZQA found that "most candidates had successfully answered the question and did not check their answer against the diagram".

"The panel found nothing in the data to suggest that the error in Question 2(d) had a negative impact on the overall performance of this cohort of students," it said.

"The panel was assured that any concerns relating to the error were addressed in the marking process; for example, any crossed-out work in candidates' responses to Question 2(d) was seen and marked."

No one picked up an inconsistency between results of an equation and values shown on a diagram in the level 2 maths NCEA exam. Photo / File

NZQA said it "takes any error in an examination extremely seriously and will implement the review panel's recommendations as a priority to ensure we provide the highest quality error-free examinations to students".

NZQA deputy chief executive Kristine Kilkelly said the error "was not due to any one individual".

"The independent review panel found that the current examination development process has an adequate number of required checks in place, and that the process was followed in this instance."

NZ Association of Maths Teachers president Gillian Frankcom-Burgess said she was happy that exams would be subject to even more stringent checking than previously.

"While it may be hoped that the examinations have no mistakes, it seems unlikely that all mistakes will always be picked up."