There seem to be so many different types of evaluation and a large number of ways in which they can be used. I'm sure I've read and re-read the Wiki site a dozen times over the past few days and tried to blog my thoughts each time...yet nothing came out.
Our task for these weeks were to:
- Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
- Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.
BUT!
I've finally decided to forget the project for now and consider the possible evaluation types that would be needed for any online project in my area. I believe this process is finally starting to make evaluation 'click' for me.
So, I shall imagine I am starting a project from scratch.
The likely stakeholders:
Manukau Institute of Technology
Hairdressing ITO
Hairdressing tutors
Hairdressing students
Reeves & Hedberg state the 4 main methods of gathering information are:
- Front end analysis (also known as needs analysis): This should be performed first to determine whether or not the project will be of benefit to the stakeholders and if so, the nature of their needs as well as their perceptions and expectations.
- Formative evaluation: This should be performed during the development and implementation stages of the project to identify any modifications which may be necessary.
- Summative evaluation: Usually carried out on completion of the project to determine the impact and whether or not the outcomes have been reached
- Monitoring or integrative evaluation (also known as maintenance evaluation): Used to determine the extent and impact the project has had on teaching and learning as well as its integration into teaching and learning activities.
As it's likely I'll be starting from scratch, all four of the above methods would be important.
But what type of paradigms and models would I use?
Front end analysis:
More than likely 'Analytic-Empirical-Positivist-Quantitative Paradigm' (Reeves, 1996).
I'm still deciding on the model that would be best for this and I'm keen to research other models (if there are any?) .
My thinking is that this analysis would evaluate the viability of the project going ahead and include projected cost vs ROI vs stakeholder support vs stakeholder needs vs stakeholder wants - ideally it would be quantitative/qualitative.
Formative evaluation:
I would suggest Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would work best in this context.
The appropriate model would be 'Multiple Methods Evaluation' (Mark and Shotland 1987). Seems to me, if one wants to ensure the final product will work well then multiple methods of evaluation should be used. At this stage it would be imperative to gain quantitative and qualitative feedback from the end users (both students and staff) which would allow for triangulation and (I believe) would provide a reasonable indication of the likely outcomes.
Summative evaluation:
Once again I believe Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would be the best fit here.
I believe the ideal model would be 'Experimental Evaluation' (Schuman, 1967). I suspect this could be time consuming and difficult to conduct and analyse but to my mind it would provide a valid comparison of whether or not the project worked (...and possibly how/why it worked?)
???Question here - something I've always wondered about the use of a 'control group' in an educational setting...if one group performs noticeably better than the other then has the group with lower performance been disadvantaged?
I would imagine there would have to be some very clear guidelines and student agreement if using a control group.
Monitoring or integrative evaluation:
See my thoughts on 'Formative Evaluation' (above). I think the same techniques would be appropriate and would provide information on possible changes which need to be made.
Phew!
It feels good to post this and I hope I'm finally 'clicking' when it comes to evaluation paradigms and models.
I shall now womble off and try to find some articles :D
2 comments:
Hi Sue
Your question about control groups and possibly putting students at a disadvantage reminds me of a topic in HR I used to teach. We would look at a range of industrial experiments that attempted to guide HR practice - one was called the Hawthorne Experiment. Basically, some Harvard researchers were carrying out experiments in changing the workplace environment and measuring any differences in performance. For each change made there was always an increase in performance. This led to a number of theories (http://en.wikipedia.org/wiki/Hawthorne_effect) but generally it was concluded that people tend to try harder when they are being observed! This made me think about a control group in an education setting - are they disadvantaged or are the group in the experiment just responding to greater interest in their learning?? Both groups have the same educational opportunity but one is more motivated to succeed. This then makes me wonder about the validity of doing this kind of research - any perceivable difference in performance may be put down to whatever is being evaluated when it might just be the improved efforts of the students during the experiment. My head hurts now :)
Yvonne
Sue good to see you making the decision to concentrate on understanding the types of evaluation before deciding on your project. I really like the way you have laid out the paradigms and models under the types of evaluation.
Do not forget to consider evaluation types such as usability (formative) and expert review (formative and summative)- Remember that summative evaluation also includes effectiveness and impact types of evaluation.
An Experimental Evaluation model would fit best under the Analytic-Empirical-Positivist-Quantitative Paradigm. Hedberg and Reeves (2003) do not recommend an experimental approach when measuring the effectiveness or impact of an interactive learning system such as an online course or multimedia product. Apparently it is extremely difficult to keep all the factors constant for the participants in both the control group and the experimental group. Experimental methods can be done as part of a usability study and could be incorporated into a mixed-methods approach.
I really like Reeves and Hedberg's explanation about the difference between formative and summative evaluation. "Formative 'improves' what is being evaluated", and summative evaluation is done to 'prove' how effective or worthwhile something is.
In terms of thinking about a project - what sort of resources and online courses are being used in hairdressing which you could investigate - learners or staff satisfaction, perceptions of the experience?
Post a Comment