I'm still here :)

Hello fellow Evaluation bloggers,

A quick post to let you all know I am still here - just been tied up with work commitments and a horrible cold over the last week.
Hopefully I will get to respond to your comments and finish my plan and presentation over the coming week.

Sue

Sue’s Evaluation Plan - Draft

Ummm...seems I need to make some time to familiarise myself with Google Docs.
This wasn't quite what I expected but it will have to do for now :)

Many thanks to Hilary for letting me know the link in my previous post could not be accessed - hope you are all OK with reading this as a Blog post.

-------

Introduction: [This section introduces the major sections of the plan as well as the primary people involved in writing the plan.]

This plan describes the background, purposes, limitations, audiences, decisions, methods, sample, instrumentation, procedures, logistics and timeline for an evaluation of a formative assessment of Unit 2873 - Demonstrate knowledge of hairdressing trichology; Element 1 - Describe functions of the skin.

The author is the writer of this plan.

Background: [This section describes any information which is needed to provide the reader with an understanding of the background of the interactive multimedia that is being evaluated.]

The School of Hair & Beauty at Manukau Institute of Technology has been using summative online assessment for the theory elements of Level 2 Hairdressing unit standards, for approximately 5 years.

These assessments are generated using the ‘Test function’ of BlackBoard and are conducted under assessment conditions, in a computer lab, during timetabled hours. Question types utilized for this level are predominately Multiple Choice, Multiple Answer and Matching.

As per the assessment policy of this School, students are permitted three attempts at completing an assessment.

Anecdotal evidence over the past 5 years suggests the hairdressing students prefer online assessment over other paper-based written assessments - mainly because they obtain instant feedback. Students have also verbally commented they initially thought the assessments would be “easy” because of the question types used but also realized they could not be considered competent if the knowledge was not known or understood.

The author has also had positive verbal feedback from mature students (aged 30 and over) that being forced to do this type of assessment has given them more confidence in computers, and using computers.

Anecdotal evidence also suggests tutors prefer this type of assessment as it reduces marking time. However, there has also been much discussion over the logistics of organizing computer labs and invigilators for resubmissions of incomplete assessments, as well as providing assessment opportunities for those students who did not attend the timetabled assessment.

To date, formative assessments have been paper based may not reflect the online environment. The author is not clear on why this is so but anecdotal evidence would suggest the perceived time and knowledge in creating an online formative assessment, (which emulates the summative assessment) vs the student outcome could be worth evaluating.

Purposes: [This section thoroughly describes the purposes of the evaluation. A single plan can address a variety of purposes, but all must be delineated clearly. Evaluation is always a political process and all parties must accept the purposes for the evaluation to be successful.]

This evaluation will establish the effectiveness of online formative assessment by:

Student evaluation

• analyzing student engagement in the online formative evaluation process

• analyzing the average number of resits students need to complete the summative assessment

• comparing results from previous years, or classes, who did not have access to the online formative assessment

Tutor evaluation:

• analyzing the perceived time in creating the formative assessment

• analyzing the perceived effectiveness of the formative assessment

Limitations: [This section spells out any limitations to the interpretation and generalizability of the evaluation. It should also describe potential threats to the reliability and validity of the evaluation design and instrumentation.]

It is intended that this evaluation will include student and tutor feedback. It is proposed that HITO and industry feedback would be beneficial although this may be limited, or unobtainable due to time constraints

The validity and reliability of this evaluation will also depend on the availability of past results and their interpretation.

Audiences: [This section specifies all the primary and secondary audiences or consumers of the evaluation. In general, it is recommended to open the evaluation up to as many people or agencies as the client will allow.]

It is intended that this evaluation will include student and tutor feedback. It is proposed that HITO and industry feedback would be beneficial although this may be limited due to time constraints (see above).

Decisions: [This section is probably the most difficult, but it should be included if the evaluation is to have meaningful impact on decision-making. Trying to anticipate the decisions which can be influenced by an evaluation takes creativity and trust. Many developers do not wish to anticipate negative outcomes for their efforts, but these too must be considered.]

1. Student perspective: It is hoped that the advantages and disadvantages of online formative assessments will be clarified. Specifically:

• The importance of online formative assessments to students

• Whether or not online formative assessments improve the results of online summative assessments

• Ease of access to online formative assessments

2. Tutor perspective: It is hoped the advantages and disadvantages of online formative assessments will be clarified. Specifically:

• The importance of online formative assessments to tutors

• Whether or not online formative assessments improve the results of online summative assessments

• Ease of developing online formative assessments

• Ease of interpreting online formative assessments and gauging how ready the student is for summative assessment

Questions: [A key element of a sound evaluation plan is careful specification of the questions to be addressed by the evaluation design and data collection methods. The clearer and more detailed these questions are, the more likely that you will be able to provide reliable and valid answers to them.]

(Still working on the questions - to follow)

Methods: [ This section describes the evaluation designs and procedures. There are scores of designs and hundreds of procedures which can be used. The keys to success are matching these options to the purposes and questions of your client and keeping within the budget and time line of the study.]

(Still working on the procedures - to follow)

Sample: [This section specifies exactly which students, trainers, and other personnel will participate in the evaluation. If necessary, a rationale for sample sizes should also be included.]

Students: It is proposed that >30, <60>

Tutors: It is proposed that 5 tutors will participate in the evaluation.

Instrumentation: [This section describes all the evaluation instruments and tools to be used in the evaluation. Actual instruments should be included in appendices for review and approval. ]

(Still working on the evaluation instruments - to follow)

Logistics: [This section spells out who will be responsible for the various implementation, analysis, and reporting aspects of the evaluation.]

The author is responsible for implementation, analysis and reporting of this evaluation.

Time Line: [This section presents the schedule for implementation, analysis, and reporting of the evaluation. ]

(Still working on the Time line - to follow)

Budget: [This section "costs out" the finances for the evaluation. Personnel time usually is the major cost factor. Evaluators often charge from two hundred to several thousand dollars per day depending on their expertise and reputation. Other significant cost factors are travel, data preparation (e.g., transcribing taped interviews), and document duplication. ]

(Still working on the Budget - to follow)

Draft Evaluation Plan

I've completed my first draft of my evaluation plan, *phew*!
I still have to work out the Questions, Methods, Evaluation Instruments, Timeline and Budget....and of course spell check and grammar check :)

If you want to see where I'm at then click the following link Sue's Draft Evaluation Plan

Any and all feedback is very much welcome :)

Do You Sometimes Wonder If...

The Web is out to get you?

I had some time this weekend to get my plan written.
Bronwyn's feedback suggested I should rethink/relook at my second guideline to ensure it was relevant to my evaluation.
I totally agree - but the site has been down yesterday and today.

Fingers crossed it will be fixed tomorrow (Monday).
In the meantime I shall keep writing my plan...

Weeks 7 & 8

At least I think it's weeks 7 & 8 - to be honest I've lost count of the weeks.

This week I've been trying to decide what I will evaluate so I sent an e-mail to Bronwyn to clarify a few things.
Bronwyn asked if I could post my ideas on my Blog. I think it's easiest to post my e-mail:
-----
From Tue, Apr 22, 2008:

Hi Bronwyn,

I'm thinking ahead here to catch up, so hope the following makes sense...

Can you please clarify re the evaluation plan and evaluation we need to conduct:
- Can this be on anything we are currently using 'online'?

Right now our only 'online' teaching/learning resources are Level 2 summative theory assessments for hairdressing. I say 'online' (with inverted commas) because while these are via eMIT/BlackBoard they are are conducted under assessment conditions, at MIT, during timetabled hours and are not accessible outside those times.

We have just introduced an online formative assessment, for one unit, which the students can access at any time.

So further questions:
Can I base my plan, presentation and evaluation around the effectiveness of online L2 summative assessments for hairdressing students? (I could, say, compare the feedback of the students and tutors who experienced this in Year One and who then went to written assessments in Year Two (Level 3-4 assessments).

Or - as another idea, as we have just introduced a formative assessment:
Could I perhaps compare the results of those students who took advantage of the formative assessment, with those who did not?

For both of the above, I should be able to gather feedback from students, staff, probably the Hairdressing ITO...possibly industry if there is time.

Many thanks in advance for your thoughts,

Sue

Weeks Five & Six - It's nice when things start to 'click'

I've been 'stuck' with my learning during weeks 5 & 6.
There seem to be so many different types of evaluation and a large number of ways in which they can be used. I'm sure I've read and re-read the Wiki site a dozen times over the past few days and tried to blog my thoughts each time...yet nothing came out.

Our task for these weeks were to:

• Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
• Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

I think I've been 'stuck' in the "your project" aspect. As yet I have no idea what my project will be and therefore have found it difficult to consider the evaluation types that would be necessary.

BUT!
I've finally decided to forget the project for now and consider the possible evaluation types that would be needed for any online project in my area. I believe this process is finally starting to make evaluation 'click' for me.

So, I shall imagine I am starting a project from scratch.
The likely stakeholders:
Manukau Institute of Technology
Hairdressing ITO
Hairdressing tutors
Hairdressing students

Reeves & Hedberg state the 4 main methods of gathering information are:
- Front end analysis (also known as needs analysis): This should be performed first to determine whether or not the project will be of benefit to the stakeholders and if so, the nature of their needs as well as their perceptions and expectations.

- Formative evaluation: This should be performed during the development and implementation stages of the project to identify any modifications which may be necessary.

- Summative evaluation: Usually carried out on completion of the project to determine the impact and whether or not the outcomes have been reached

- Monitoring or integrative evaluation (also known as maintenance evaluation): Used to determine the extent and impact the project has had on teaching and learning as well as its integration into teaching and learning activities.

As it's likely I'll be starting from scratch, all four of the above methods would be important.

But what type of paradigms and models would I use?
Front end analysis:
More than likely 'Analytic-Empirical-Positivist-Quantitative Paradigm' (Reeves, 1996).
I'm still deciding on the model that would be best for this and I'm keen to research other models (if there are any?) .
My thinking is that this analysis would evaluate the viability of the project going ahead and include projected cost vs ROI vs stakeholder support vs stakeholder needs vs stakeholder wants - ideally it would be quantitative/qualitative.

Formative evaluation:
I would suggest Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would work best in this context.
The appropriate model would be 'Multiple Methods Evaluation' (Mark and Shotland 1987). Seems to me, if one wants to ensure the final product will work well then multiple methods of evaluation should be used. At this stage it would be imperative to gain quantitative and qualitative feedback from the end users (both students and staff) which would allow for triangulation and (I believe) would provide a reasonable indication of the likely outcomes.

Summative evaluation:
Once again I believe Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would be the best fit here.
I believe the ideal model would be 'Experimental Evaluation' (Schuman, 1967). I suspect this could be time consuming and difficult to conduct and analyse but to my mind it would provide a valid comparison of whether or not the project worked (...and possibly how/why it worked?)

???Question here - something I've always wondered about the use of a 'control group' in an educational setting...if one group performs noticeably better than the other then has the group with lower performance been disadvantaged?
I would imagine there would have to be some very clear guidelines and student agreement if using a control group.

Monitoring or integrative evaluation:
See my thoughts on 'Formative Evaluation' (above). I think the same techniques would be appropriate and would provide information on possible changes which need to be made.

Phew!
It feels good to post this and I hope I'm finally 'clicking' when it comes to evaluation paradigms and models.

I shall now womble off and try to find some articles :D

Week four: Evaluation paradigms and models

Must admit I was thankful for Bronwyn's definition of paradigm this week.
Bronwyn said:

"The word paradigm conjures up visions of stiff and starchy academia doesn't it? Well think of it as a style or belief. You will all have heard the words paradigm-shift when people talk about how eLearning is changing how we teach; meaning the change from teacher-centred to student-centred learning."

Previously, I blogged about a 'brain shift'...was I talking about a similar thing?
I gather, in my words, it essentially means 'a model, or pattern'. Therefore "paradigm-shift' means using a different model or pattern...and may require a "brain shift' to do so?

Then in the readings (see link below) there was the word Hermeneutic! I had to look that one up! Grief! (should also add - all good, I enjoy word puzzles)
Seems it means: 'Interpretive; explanatory' (dictionary.com)

According to Professor Tom Reeves there are four Educational Paradigms:

1. Analytic-Empirical-Positivist-Quantititive Paradigm: the most established of the paradigms. My interpretation is this is something that can be measured, such as a set of statistics. Supposedly objective...but I do wonder if the viewpoint of the surveyor has an impact here...
   
2. Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm : In my words - this is the analysis of comments regarding how we (humans) do things. I still don't really understand this but hope I'm on the right track...apparently subjective.
   
3. Critical Theory-Neomarxist-Postmodern-Praxis Paradigm: Reeves suggests proponents of this paradigm are 'social activists' and "view themselves as the forces of liberation engaged in mortal conflict with the powers of liberation". Must admit I'm not too sure about this one and would need to research more before making a comment but my gut feeling is that it sounds far to political for me.
   
4. Eclectic-Mixed methods-Pragmatic Paradigm: I'm still reading and re-reading this. Reeves suggests this is the "most useful" paradigm. And I gather it is a mix of the above...? (source: paradigms by Professor Tom Reeves.)

While I'm still digesting all of the above, I wondering about the impact technology has has on our lives...and how we measure that impact.

I think I need to stop thinking :)