Sue’s Evaluation Plan - Draft

Ummm...seems I need to make some time to familiarise myself with Google Docs.
This wasn't quite what I expected but it will have to do for now :)

Many thanks to Hilary for letting me know the link in my previous post could not be accessed - hope you are all OK with reading this as a Blog post.

-------

Introduction: [This section introduces the major sections of the plan as well as the primary people involved in writing the plan.]

This plan describes the background, purposes, limitations, audiences, decisions, methods, sample, instrumentation, procedures, logistics and timeline for an evaluation of a formative assessment of Unit 2873 - Demonstrate knowledge of hairdressing trichology; Element 1 - Describe functions of the skin.

The author is the writer of this plan.

Background: [This section describes any information which is needed to provide the reader with an understanding of the background of the interactive multimedia that is being evaluated.]

The School of Hair & Beauty at Manukau Institute of Technology has been using summative online assessment for the theory elements of Level 2 Hairdressing unit standards, for approximately 5 years.

These assessments are generated using the ‘Test function’ of BlackBoard and are conducted under assessment conditions, in a computer lab, during timetabled hours. Question types utilized for this level are predominately Multiple Choice, Multiple Answer and Matching.

As per the assessment policy of this School, students are permitted three attempts at completing an assessment.

Anecdotal evidence over the past 5 years suggests the hairdressing students prefer online assessment over other paper-based written assessments - mainly because they obtain instant feedback. Students have also verbally commented they initially thought the assessments would be “easy” because of the question types used but also realized they could not be considered competent if the knowledge was not known or understood.

The author has also had positive verbal feedback from mature students (aged 30 and over) that being forced to do this type of assessment has given them more confidence in computers, and using computers.

Anecdotal evidence also suggests tutors prefer this type of assessment as it reduces marking time. However, there has also been much discussion over the logistics of organizing computer labs and invigilators for resubmissions of incomplete assessments, as well as providing assessment opportunities for those students who did not attend the timetabled assessment.

To date, formative assessments have been paper based may not reflect the online environment. The author is not clear on why this is so but anecdotal evidence would suggest the perceived time and knowledge in creating an online formative assessment, (which emulates the summative assessment) vs the student outcome could be worth evaluating.

Purposes: [This section thoroughly describes the purposes of the evaluation. A single plan can address a variety of purposes, but all must be delineated clearly. Evaluation is always a political process and all parties must accept the purposes for the evaluation to be successful.]

This evaluation will establish the effectiveness of online formative assessment by:

Student evaluation

• analyzing student engagement in the online formative evaluation process

• analyzing the average number of resits students need to complete the summative assessment

• comparing results from previous years, or classes, who did not have access to the online formative assessment

Tutor evaluation:

• analyzing the perceived time in creating the formative assessment

• analyzing the perceived effectiveness of the formative assessment

Limitations: [This section spells out any limitations to the interpretation and generalizability of the evaluation. It should also describe potential threats to the reliability and validity of the evaluation design and instrumentation.]

It is intended that this evaluation will include student and tutor feedback. It is proposed that HITO and industry feedback would be beneficial although this may be limited, or unobtainable due to time constraints

The validity and reliability of this evaluation will also depend on the availability of past results and their interpretation.

Audiences: [This section specifies all the primary and secondary audiences or consumers of the evaluation. In general, it is recommended to open the evaluation up to as many people or agencies as the client will allow.]

It is intended that this evaluation will include student and tutor feedback. It is proposed that HITO and industry feedback would be beneficial although this may be limited due to time constraints (see above).

Decisions: [This section is probably the most difficult, but it should be included if the evaluation is to have meaningful impact on decision-making. Trying to anticipate the decisions which can be influenced by an evaluation takes creativity and trust. Many developers do not wish to anticipate negative outcomes for their efforts, but these too must be considered.]

1. Student perspective: It is hoped that the advantages and disadvantages of online formative assessments will be clarified. Specifically:

• The importance of online formative assessments to students

• Whether or not online formative assessments improve the results of online summative assessments

• Ease of access to online formative assessments

2. Tutor perspective: It is hoped the advantages and disadvantages of online formative assessments will be clarified. Specifically:

• The importance of online formative assessments to tutors

• Whether or not online formative assessments improve the results of online summative assessments

• Ease of developing online formative assessments

• Ease of interpreting online formative assessments and gauging how ready the student is for summative assessment

Questions: [A key element of a sound evaluation plan is careful specification of the questions to be addressed by the evaluation design and data collection methods. The clearer and more detailed these questions are, the more likely that you will be able to provide reliable and valid answers to them.]

(Still working on the questions - to follow)

Methods: [ This section describes the evaluation designs and procedures. There are scores of designs and hundreds of procedures which can be used. The keys to success are matching these options to the purposes and questions of your client and keeping within the budget and time line of the study.]

(Still working on the procedures - to follow)

Sample: [This section specifies exactly which students, trainers, and other personnel will participate in the evaluation. If necessary, a rationale for sample sizes should also be included.]

Students: It is proposed that >30, <60>

Tutors: It is proposed that 5 tutors will participate in the evaluation.

Instrumentation: [This section describes all the evaluation instruments and tools to be used in the evaluation. Actual instruments should be included in appendices for review and approval. ]

(Still working on the evaluation instruments - to follow)

Logistics: [This section spells out who will be responsible for the various implementation, analysis, and reporting aspects of the evaluation.]

The author is responsible for implementation, analysis and reporting of this evaluation.

Time Line: [This section presents the schedule for implementation, analysis, and reporting of the evaluation. ]

(Still working on the Time line - to follow)

Budget: [This section "costs out" the finances for the evaluation. Personnel time usually is the major cost factor. Evaluators often charge from two hundred to several thousand dollars per day depending on their expertise and reputation. Other significant cost factors are travel, data preparation (e.g., transcribing taped interviews), and document duplication. ]

(Still working on the Budget - to follow)