Moderation
Introduction
The UniSA Assessment Policy and Procedures Manual clause 4.3 refers to assessment methods operating in a valid and reliable framework whilst acknowledging that human judgment is a significant element in the process. It also identifies that moderation is an important part of the validity and reliability of assessment activities. The manual determines that:
- "Moderation (including external moderation where appropriate) of the setting of assessment tasks and of marking will be established to improve the validity and reliability of assessment processes."
While the manual does not provide a definition of moderation, a working understanding can be framed as follows:
- Moderation is a quality review and assurance process which supports the assessment setting and marking activities of course coordinators and their staff. It involves using other academics and qualified staff to perform tailored tasks to ensure that assessment tasks and marking are valid and reliable. Essentially, it is a checking process.
Operation
Moderation is an internal control activity and, therefore, is a function of a cost/benefit approach. It cannot and is not designed to provide absolute assurance about the veracity of a particular assessment activity. It is there to provide a reasonable level of assurance that assessment activities have been undertaken appropriately and that there is a reasonable level of confidence of the results provided to students. There are generally a number of elements that affect the levels of confidence in a moderation process. Some of these are:
- the length of time available
- the financial resources
- availability of qualified staff
- the location, delivery mode or other unique issues relating to the course. For example, the level of moderation confidence for an offshore course, delivered entirely through an agent, will probably be higher than an internal course wholly delivered by a course coordinator in a face-to-face environment
- unique issues that arise from time-to-time (eg. contractual requirements)
Staff should consider the levels of confidence they are after. This does not mean that the level needs to be quantified, but there does need to be some consideration and resolution about the the objectives of the moderation task.
Some moderation processes
Moderation essentially operates at two levels:
- at the assessment setting stage
- after marking (and before processing of final results)
The following section looks at these two levels and provides some ideas for course coordinators. The list should not be seen as exhaustive and moderation processes should be tailored to particular circumstances.
Assessment setting
This is probably the easiest of the two moderation processes. A number of strategies exist here.
- Ask an appropriate academic colleague to review and provide critical feedback on various assessment tasks at the design stage. Depending on the particular assessment, many academic moderators will undertake the assessment as if they were a student (particularly relevant to exams).
- Where there are multiple academics involved in the teaching and delivery (eg. large classes), it may be useful for course coordinators to call a meeting of all academic staff involved in the course to discuss a particular assessment task. A draft solutions guide and marking schema can be presented and the group can analyse, critique and improve where possible the assessment task.
The aim of moderating assessment tasks before students are asked to undertake them, is to address any ambiguities, vagueness, capacities, length, etc., issues which might cause concern to students.
After marking
After an assessment task has been marked there will often be a need to conduct some level of moderation on the results. The variety and degree of tasks can be considerable and will be a function of the objectives a course coordinator has determined. The following, however, might provide some assistance in framing those tasks:
- Moderate at the margins - that is around the specific mark levels of particular grades (eg. 50%, 55%, 65% and so on). Here you would select a sufficient number of scripts around these marks and review them for the appropriateness of the given mark and, therefore, the grade. This process gives confidence about the grading of assessment.
- If you have multiple markers, examine a distribution of the marks awarded by the various markers (and if there are multiple questions, then for each question) to ascertain if there are markers who might lie outside of the average or a general trend.
- Conduct all marking at the same time and the same place. This does allow for a concerted effort to address any issues that might be raised by markers in a consistent manner. However, course coordinators can also conduct a moderation (or audit) process of marked papers during the marking. This gives a high level of confidence about the marking and results but is resource intensive and relies on getting all the markers together (which can obviously be difficult).
- Random sampling by the course coordinator of the whole population of papers and selecting a sufficient number to meet the particular confidence level. This process will address all marks and not just grades. This is a relatively straight-forward exercise if all papers are readily available and in the control of the course coordinator. However, circumstances can exist where this can be compromised - for example, where papers are in an offshore location. If such circumstances do exist then course coordinators need to consider how they can achieve their confidence levels whilst dealing with those peculiarities.
What happens if the moderation finds a problem?
This can be a difficult question to address in some some circumstances. While it is hoped that moderation supports the assessment activities, occasions will arise where the moderation will detect a problem. This is more of a concern at the marking stage rather than the assessment setting stage.
If a problem in marking is detected it cannot be ignored. Some form of action needs to undertaken but it will be up to course coordinators as to how they deal with it as it will be a function of the unique matter confronting them. Some areas of guidance are:
- Consider taking another sample. Your first sample may not have been representative of the population. After moderating the next sample, review the results and decide where the problem or error lay.
- Identify if there is a particular issue that the moderation has identified. For example, it may only be one question in an exam and the amount of follow-up work needed to be done might not be large. It is important to clearly identify the size and scale of the problem so subsequent action is directed at the specific issue.
If it happens that the moderation has unearthed a serious issue in the marking process and the course coordinator is faced with time or resource constraints in addressing it, staff should consult their relevant Program Director and/or Head of School to discuss the various options that might be available.