Proposal
Table of Contents
- The Rationale for and Aims of the Project
- Relationship to Research in the Area
- Current Status of the Proposed Project
- The Approach the Project Will Take
- The Anticipated Deliverables and Outcomes
- Conclusion
- Attachment 1 - Bibliography
- Attachment 3 - Time lines
- Attachment 4 - Project Team
- Attachment 5 - Screenshots of Prototype
- Attachment 6 - Proposed System Architecture
- Attachment 7 - Evaluation Plan
- Attachment 8 - Dissemination Plan
- Attachment 9 - Relationship to the project led by Dr Jo McKenzie (UTS)
Current Status of the Proposed Project
This project builds on the experience gained by the project team leader and two members of the proposed project team from a project that led to the design and development of a prototype of a peer review tool. The prototype has been trialed at the University of South Australia and the results of the trials published (George, Wood and Wache, 2004). While limited in scope and functionality, the findings from the trials together with the enthusiastic response from the higher education community at conference presentations, suggest the proposed system when fully implemented as an open source tool will be well received. The prototype on which this proposed system will be based is described briefly in the following section. The prototype can be viewed online at the following url: http://students.unisa.cc/~uninet.wooddl/public/peer/index.htm.
The tool is constructed around four sets of considerations: instructional design, interface design, the use of multimedia to engage learners and the technical aspects of interactive educational multimedia. These areas have been developed through consideration of the literature. Several of these major sections are broken down into smaller subsections for ease of consideration:
- instructional design
- clarity of expectations
- building student knowledge
- learning activities
- assessment
- evaluation
- human interaction
- interface design
- use of media
- interactive multimedia
- writing style and accuracy of text
- copyright
- technical aspects
In each of these sections (or sub-sections), items were developed. These items:
- directly relate to quality concerns agreed in the literature
- are expressed in non-technical ways
- use the same language and constructs as checklists developed for other areas of teaching such as lecturing
- incorporate an easy-to-use ratings system
- provide for qualitative feedback through comments
The rating system used to measure the extent to which the specified metrics meet these criteria is a 5-point Likert scale ranging from strongly agree to strongly disagree for metrics that involve value judgments, and from always to never for metrics that consider the frequency of occurrence (see Figure 1 in Attachment 5). Each criterion also provides a free form text area for comments since a combination of quantitative (Likert rating scale) and qualitative (open-ended user comments) measures will most likely yield comprehensive results (Laycock and Nowlan, 2000).
Educative Aspects
The educative dimension is central to both the ‘just-in-time’ approach to professional development and approaches which involve more formal educational development. Each of the items has a ‘more information’ button which is a hyperlink to detailed information about the issue including explanations and references to the literature. For example, one of the criteria specified in the sub-section ‘Clarity of Expectations’ is an item referring to the statement of objectives or learning outcomes. There is often confusion among reviewers about the difference between general statements about the overall goals and clearly specified objectives. By clicking on the ‘more information’ hyperlink as shown in Figure 1 (Attachment 5), the reviewer can check their understanding of these terms and also learn more about effective techniques for specifying objectives or learning outcomes from the hyperlink references included in the related explanatory screen (Figure 2, Attachment 5).
Inclusivity
Items addressing issues of inclusivity have been embedded across the four sections of the tool. These relate to socio-cultural aspects including gender and culture, and have a particular focus on W3C Techniques for Web Content Accessibility Guidelines. The decision to embed these rather than to extract them into separate categories is based on the view that essentially the items reflect good teaching and ought to be seen in a more integrated way. For example, a course that includes streaming media presentations containing voice-over or dialogue would need to be reviewed to determine whether the media is accessible to students with hearing impairments. Information on the accessibility issues for people with disabilities and the relevant W3C checkpoints can be accessed by selecting the ‘more information’ link next to the criterion relating to synchronised captions for video clips that contain audio tracks in the ‘Use of Media’ section of the tool (Figure 3, Attachment 5).
Flexibility
One of the issues around the development of the tool has been to ensure it has maximum flexibility. For this reason, the tool is very detailed and may be used in part or as a whole depending on the intended outcome. These may include:
- course development or improvement through personal reflection
- formative or summative evaluation of online materials by peers
- career advancement through providing evidence that supports applications for promotion, tenure or awards
- framework for professional development staff to use when working with academics.
Limitations of Prototype and Proposed Enhancements
The current embodiment of the tool is limited in the following ways:
- The number of categories included in the prototype and the range of criteria addressed within each of the categories is not comprehensive. Project team members will assist in the development of new categories and the addition of more extensive criteria within each of the categories.
- Not all criteria have been populated with links to detailed information about the criterion including explanations and references to the literature. Project team members will contribute by sharing relevant information and inviting academics within their respective institutions to also contribute to the population of these links.
- The website is not linked to a database, and the Likert-scales are static and non-functional. This means that the only way to use the prototype is by printing out the scales and completing the review by hand. In the proposed redevelopment, the peer review will be completed online and results of the review stored in a data base for retrieval by academics in support of their applications for promotion and awards (see Attachment 6 illustrating proposed system architecture).
- There is currently no facility enabling academics to populate the tool with their own criteria. The proposed tool will incorporate a wizard front-end enabling them to select from banks of criteria, and add their own categories of criteria to tailor the peer review survey to best suit the nature of their specific courses to be reviewed (see Attachment 6, which illustrates the proposed functionality of the Wizard).
- There are as yet no case studies demonstrating best practice relating to the categories incorporated into the tool. Project team members, through their extensive networks, will contribute to these case studies. Provision will be made via the linked website of resources for academics not associated with the project to contribute their own case studies by registering their interest through an interactive form (Attachment 6 part B).