Weeks 7 & 8

At least I think it's weeks 7 & 8 - to be honest I've lost count of the weeks.

This week I've been trying to decide what I will evaluate so I sent an e-mail to Bronwyn to clarify a few things.
Bronwyn asked if I could post my ideas on my Blog. I think it's easiest to post my e-mail:
-----
From Tue, Apr 22, 2008:

Hi Bronwyn,

I'm thinking ahead here to catch up, so hope the following makes sense...

Can you please clarify re the evaluation plan and evaluation we need to conduct:
- Can this be on anything we are currently using 'online'?

Right now our only 'online' teaching/learning resources are Level 2 summative theory assessments for hairdressing. I say 'online' (with inverted commas) because while these are via eMIT/BlackBoard they are are conducted under assessment conditions, at MIT, during timetabled hours and are not accessible outside those times.

We have just introduced an online formative assessment, for one unit, which the students can access at any time.

So further questions:
Can I base my plan, presentation and evaluation around the effectiveness of online L2 summative assessments for hairdressing students? (I could, say, compare the feedback of the students and tutors who experienced this in Year One and who then went to written assessments in Year Two (Level 3-4 assessments).

Or - as another idea, as we have just introduced a formative assessment:
Could I perhaps compare the results of those students who took advantage of the formative assessment, with those who did not?

For both of the above, I should be able to gather feedback from students, staff, probably the Hairdressing ITO...possibly industry if there is time.

Many thanks in advance for your thoughts,

Sue

Weeks Five & Six - It's nice when things start to 'click'

I've been 'stuck' with my learning during weeks 5 & 6.
There seem to be so many different types of evaluation and a large number of ways in which they can be used. I'm sure I've read and re-read the Wiki site a dozen times over the past few days and tried to blog my thoughts each time...yet nothing came out.

Our task for these weeks were to:

• Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
• Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

I think I've been 'stuck' in the "your project" aspect. As yet I have no idea what my project will be and therefore have found it difficult to consider the evaluation types that would be necessary.

BUT!
I've finally decided to forget the project for now and consider the possible evaluation types that would be needed for any online project in my area. I believe this process is finally starting to make evaluation 'click' for me.

So, I shall imagine I am starting a project from scratch.
The likely stakeholders:
Manukau Institute of Technology
Hairdressing ITO
Hairdressing tutors
Hairdressing students

Reeves & Hedberg state the 4 main methods of gathering information are:
- Front end analysis (also known as needs analysis): This should be performed first to determine whether or not the project will be of benefit to the stakeholders and if so, the nature of their needs as well as their perceptions and expectations.

- Formative evaluation: This should be performed during the development and implementation stages of the project to identify any modifications which may be necessary.

- Summative evaluation: Usually carried out on completion of the project to determine the impact and whether or not the outcomes have been reached

- Monitoring or integrative evaluation (also known as maintenance evaluation): Used to determine the extent and impact the project has had on teaching and learning as well as its integration into teaching and learning activities.

As it's likely I'll be starting from scratch, all four of the above methods would be important.

But what type of paradigms and models would I use?
Front end analysis:
More than likely 'Analytic-Empirical-Positivist-Quantitative Paradigm' (Reeves, 1996).
I'm still deciding on the model that would be best for this and I'm keen to research other models (if there are any?) .
My thinking is that this analysis would evaluate the viability of the project going ahead and include projected cost vs ROI vs stakeholder support vs stakeholder needs vs stakeholder wants - ideally it would be quantitative/qualitative.

Formative evaluation:
I would suggest Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would work best in this context.
The appropriate model would be 'Multiple Methods Evaluation' (Mark and Shotland 1987). Seems to me, if one wants to ensure the final product will work well then multiple methods of evaluation should be used. At this stage it would be imperative to gain quantitative and qualitative feedback from the end users (both students and staff) which would allow for triangulation and (I believe) would provide a reasonable indication of the likely outcomes.

Summative evaluation:
Once again I believe Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would be the best fit here.
I believe the ideal model would be 'Experimental Evaluation' (Schuman, 1967). I suspect this could be time consuming and difficult to conduct and analyse but to my mind it would provide a valid comparison of whether or not the project worked (...and possibly how/why it worked?)

???Question here - something I've always wondered about the use of a 'control group' in an educational setting...if one group performs noticeably better than the other then has the group with lower performance been disadvantaged?
I would imagine there would have to be some very clear guidelines and student agreement if using a control group.

Monitoring or integrative evaluation:
See my thoughts on 'Formative Evaluation' (above). I think the same techniques would be appropriate and would provide information on possible changes which need to be made.

Phew!
It feels good to post this and I hope I'm finally 'clicking' when it comes to evaluation paradigms and models.

I shall now womble off and try to find some articles :D

Week four: Evaluation paradigms and models

Must admit I was thankful for Bronwyn's definition of paradigm this week.
Bronwyn said:

"The word paradigm conjures up visions of stiff and starchy academia doesn't it? Well think of it as a style or belief. You will all have heard the words paradigm-shift when people talk about how eLearning is changing how we teach; meaning the change from teacher-centred to student-centred learning."

Previously, I blogged about a 'brain shift'...was I talking about a similar thing?
I gather, in my words, it essentially means 'a model, or pattern'. Therefore "paradigm-shift' means using a different model or pattern...and may require a "brain shift' to do so?

Then in the readings (see link below) there was the word Hermeneutic! I had to look that one up! Grief! (should also add - all good, I enjoy word puzzles)
Seems it means: 'Interpretive; explanatory' (dictionary.com)

According to Professor Tom Reeves there are four Educational Paradigms:

1. Analytic-Empirical-Positivist-Quantititive Paradigm: the most established of the paradigms. My interpretation is this is something that can be measured, such as a set of statistics. Supposedly objective...but I do wonder if the viewpoint of the surveyor has an impact here...
   
2. Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm : In my words - this is the analysis of comments regarding how we (humans) do things. I still don't really understand this but hope I'm on the right track...apparently subjective.
   
3. Critical Theory-Neomarxist-Postmodern-Praxis Paradigm: Reeves suggests proponents of this paradigm are 'social activists' and "view themselves as the forces of liberation engaged in mortal conflict with the powers of liberation". Must admit I'm not too sure about this one and would need to research more before making a comment but my gut feeling is that it sounds far to political for me.
   
4. Eclectic-Mixed methods-Pragmatic Paradigm: I'm still reading and re-reading this. Reeves suggests this is the "most useful" paradigm. And I gather it is a mix of the above...? (source: paradigms by Professor Tom Reeves.)

While I'm still digesting all of the above, I wondering about the impact technology has has on our lives...and how we measure that impact.

I think I need to stop thinking :)

More Week 3 - Readings

The evaluation study was interesting.
I was very surprised to read that the students preferred audio over video!

Note to self:
(from the link above)

"Lessons Learned
The unanticipated results of this project strongly reinforced two lessons that can be applied to most learning technology projects: (1) the importance of actively involving the client, and (2) the importance of using proven instructional design and formative evaluation techniques."