I'm still here :)

Hello fellow Evaluation bloggers,

A quick post to let you all know I am still here - just been tied up with work commitments and a horrible cold over the last week.
Hopefully I will get to respond to your comments and finish my plan and presentation over the coming week.

Sue

Sue’s Evaluation Plan - Draft

Ummm...seems I need to make some time to familiarise myself with Google Docs.
This wasn't quite what I expected but it will have to do for now :)

Many thanks to Hilary for letting me know the link in my previous post could not be accessed - hope you are all OK with reading this as a Blog post.

-------

Introduction: [This section introduces the major sections of the plan as well as the primary people involved in writing the plan.]

This plan describes the background, purposes, limitations, audiences, decisions, methods, sample, instrumentation, procedures, logistics and timeline for an evaluation of a formative assessment of Unit 2873 - Demonstrate knowledge of hairdressing trichology; Element 1 - Describe functions of the skin.

The author is the writer of this plan.

Background: [This section describes any information which is needed to provide the reader with an understanding of the background of the interactive multimedia that is being evaluated.]

The School of Hair & Beauty at Manukau Institute of Technology has been using summative online assessment for the theory elements of Level 2 Hairdressing unit standards, for approximately 5 years.

These assessments are generated using the ‘Test function’ of BlackBoard and are conducted under assessment conditions, in a computer lab, during timetabled hours. Question types utilized for this level are predominately Multiple Choice, Multiple Answer and Matching.

As per the assessment policy of this School, students are permitted three attempts at completing an assessment.

Anecdotal evidence over the past 5 years suggests the hairdressing students prefer online assessment over other paper-based written assessments - mainly because they obtain instant feedback. Students have also verbally commented they initially thought the assessments would be “easy” because of the question types used but also realized they could not be considered competent if the knowledge was not known or understood.

The author has also had positive verbal feedback from mature students (aged 30 and over) that being forced to do this type of assessment has given them more confidence in computers, and using computers.

Anecdotal evidence also suggests tutors prefer this type of assessment as it reduces marking time. However, there has also been much discussion over the logistics of organizing computer labs and invigilators for resubmissions of incomplete assessments, as well as providing assessment opportunities for those students who did not attend the timetabled assessment.

To date, formative assessments have been paper based may not reflect the online environment. The author is not clear on why this is so but anecdotal evidence would suggest the perceived time and knowledge in creating an online formative assessment, (which emulates the summative assessment) vs the student outcome could be worth evaluating.

Purposes: [This section thoroughly describes the purposes of the evaluation. A single plan can address a variety of purposes, but all must be delineated clearly. Evaluation is always a political process and all parties must accept the purposes for the evaluation to be successful.]

This evaluation will establish the effectiveness of online formative assessment by:

Student evaluation

• analyzing student engagement in the online formative evaluation process

• analyzing the average number of resits students need to complete the summative assessment

• comparing results from previous years, or classes, who did not have access to the online formative assessment

Tutor evaluation:

• analyzing the perceived time in creating the formative assessment

• analyzing the perceived effectiveness of the formative assessment

Limitations: [This section spells out any limitations to the interpretation and generalizability of the evaluation. It should also describe potential threats to the reliability and validity of the evaluation design and instrumentation.]

It is intended that this evaluation will include student and tutor feedback. It is proposed that HITO and industry feedback would be beneficial although this may be limited, or unobtainable due to time constraints

The validity and reliability of this evaluation will also depend on the availability of past results and their interpretation.

Audiences: [This section specifies all the primary and secondary audiences or consumers of the evaluation. In general, it is recommended to open the evaluation up to as many people or agencies as the client will allow.]

It is intended that this evaluation will include student and tutor feedback. It is proposed that HITO and industry feedback would be beneficial although this may be limited due to time constraints (see above).

Decisions: [This section is probably the most difficult, but it should be included if the evaluation is to have meaningful impact on decision-making. Trying to anticipate the decisions which can be influenced by an evaluation takes creativity and trust. Many developers do not wish to anticipate negative outcomes for their efforts, but these too must be considered.]

1. Student perspective: It is hoped that the advantages and disadvantages of online formative assessments will be clarified. Specifically:

• The importance of online formative assessments to students

• Whether or not online formative assessments improve the results of online summative assessments

• Ease of access to online formative assessments

2. Tutor perspective: It is hoped the advantages and disadvantages of online formative assessments will be clarified. Specifically:

• The importance of online formative assessments to tutors

• Whether or not online formative assessments improve the results of online summative assessments

• Ease of developing online formative assessments

• Ease of interpreting online formative assessments and gauging how ready the student is for summative assessment

Questions: [A key element of a sound evaluation plan is careful specification of the questions to be addressed by the evaluation design and data collection methods. The clearer and more detailed these questions are, the more likely that you will be able to provide reliable and valid answers to them.]

(Still working on the questions - to follow)

Methods: [ This section describes the evaluation designs and procedures. There are scores of designs and hundreds of procedures which can be used. The keys to success are matching these options to the purposes and questions of your client and keeping within the budget and time line of the study.]

(Still working on the procedures - to follow)

Sample: [This section specifies exactly which students, trainers, and other personnel will participate in the evaluation. If necessary, a rationale for sample sizes should also be included.]

Students: It is proposed that >30, <60>

Tutors: It is proposed that 5 tutors will participate in the evaluation.

Instrumentation: [This section describes all the evaluation instruments and tools to be used in the evaluation. Actual instruments should be included in appendices for review and approval. ]

(Still working on the evaluation instruments - to follow)

Logistics: [This section spells out who will be responsible for the various implementation, analysis, and reporting aspects of the evaluation.]

The author is responsible for implementation, analysis and reporting of this evaluation.

Time Line: [This section presents the schedule for implementation, analysis, and reporting of the evaluation. ]

(Still working on the Time line - to follow)

Budget: [This section "costs out" the finances for the evaluation. Personnel time usually is the major cost factor. Evaluators often charge from two hundred to several thousand dollars per day depending on their expertise and reputation. Other significant cost factors are travel, data preparation (e.g., transcribing taped interviews), and document duplication. ]

(Still working on the Budget - to follow)

Draft Evaluation Plan

I've completed my first draft of my evaluation plan, *phew*!
I still have to work out the Questions, Methods, Evaluation Instruments, Timeline and Budget....and of course spell check and grammar check :)

If you want to see where I'm at then click the following link Sue's Draft Evaluation Plan

Any and all feedback is very much welcome :)

Do You Sometimes Wonder If...

The Web is out to get you?

I had some time this weekend to get my plan written.
Bronwyn's feedback suggested I should rethink/relook at my second guideline to ensure it was relevant to my evaluation.
I totally agree - but the site has been down yesterday and today.

Fingers crossed it will be fixed tomorrow (Monday).
In the meantime I shall keep writing my plan...

Weeks 7 & 8

At least I think it's weeks 7 & 8 - to be honest I've lost count of the weeks.

This week I've been trying to decide what I will evaluate so I sent an e-mail to Bronwyn to clarify a few things.
Bronwyn asked if I could post my ideas on my Blog. I think it's easiest to post my e-mail:
-----
From Tue, Apr 22, 2008:

Hi Bronwyn,

I'm thinking ahead here to catch up, so hope the following makes sense...

Can you please clarify re the evaluation plan and evaluation we need to conduct:
- Can this be on anything we are currently using 'online'?

Right now our only 'online' teaching/learning resources are Level 2 summative theory assessments for hairdressing. I say 'online' (with inverted commas) because while these are via eMIT/BlackBoard they are are conducted under assessment conditions, at MIT, during timetabled hours and are not accessible outside those times.

We have just introduced an online formative assessment, for one unit, which the students can access at any time.

So further questions:
Can I base my plan, presentation and evaluation around the effectiveness of online L2 summative assessments for hairdressing students? (I could, say, compare the feedback of the students and tutors who experienced this in Year One and who then went to written assessments in Year Two (Level 3-4 assessments).

Or - as another idea, as we have just introduced a formative assessment:
Could I perhaps compare the results of those students who took advantage of the formative assessment, with those who did not?

For both of the above, I should be able to gather feedback from students, staff, probably the Hairdressing ITO...possibly industry if there is time.

Many thanks in advance for your thoughts,

Sue

Weeks Five & Six - It's nice when things start to 'click'

I've been 'stuck' with my learning during weeks 5 & 6.
There seem to be so many different types of evaluation and a large number of ways in which they can be used. I'm sure I've read and re-read the Wiki site a dozen times over the past few days and tried to blog my thoughts each time...yet nothing came out.

Our task for these weeks were to:

• Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
• Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

I think I've been 'stuck' in the "your project" aspect. As yet I have no idea what my project will be and therefore have found it difficult to consider the evaluation types that would be necessary.

BUT!
I've finally decided to forget the project for now and consider the possible evaluation types that would be needed for any online project in my area. I believe this process is finally starting to make evaluation 'click' for me.

So, I shall imagine I am starting a project from scratch.
The likely stakeholders:
Manukau Institute of Technology
Hairdressing ITO
Hairdressing tutors
Hairdressing students

Reeves & Hedberg state the 4 main methods of gathering information are:
- Front end analysis (also known as needs analysis): This should be performed first to determine whether or not the project will be of benefit to the stakeholders and if so, the nature of their needs as well as their perceptions and expectations.

- Formative evaluation: This should be performed during the development and implementation stages of the project to identify any modifications which may be necessary.

- Summative evaluation: Usually carried out on completion of the project to determine the impact and whether or not the outcomes have been reached

- Monitoring or integrative evaluation (also known as maintenance evaluation): Used to determine the extent and impact the project has had on teaching and learning as well as its integration into teaching and learning activities.

As it's likely I'll be starting from scratch, all four of the above methods would be important.

But what type of paradigms and models would I use?
Front end analysis:
More than likely 'Analytic-Empirical-Positivist-Quantitative Paradigm' (Reeves, 1996).
I'm still deciding on the model that would be best for this and I'm keen to research other models (if there are any?) .
My thinking is that this analysis would evaluate the viability of the project going ahead and include projected cost vs ROI vs stakeholder support vs stakeholder needs vs stakeholder wants - ideally it would be quantitative/qualitative.

Formative evaluation:
I would suggest Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would work best in this context.
The appropriate model would be 'Multiple Methods Evaluation' (Mark and Shotland 1987). Seems to me, if one wants to ensure the final product will work well then multiple methods of evaluation should be used. At this stage it would be imperative to gain quantitative and qualitative feedback from the end users (both students and staff) which would allow for triangulation and (I believe) would provide a reasonable indication of the likely outcomes.

Summative evaluation:
Once again I believe Reeves 'Eclectic-Mixed Methods-Pragmatic Paradigm' would be the best fit here.
I believe the ideal model would be 'Experimental Evaluation' (Schuman, 1967). I suspect this could be time consuming and difficult to conduct and analyse but to my mind it would provide a valid comparison of whether or not the project worked (...and possibly how/why it worked?)

???Question here - something I've always wondered about the use of a 'control group' in an educational setting...if one group performs noticeably better than the other then has the group with lower performance been disadvantaged?
I would imagine there would have to be some very clear guidelines and student agreement if using a control group.

Monitoring or integrative evaluation:
See my thoughts on 'Formative Evaluation' (above). I think the same techniques would be appropriate and would provide information on possible changes which need to be made.

Phew!
It feels good to post this and I hope I'm finally 'clicking' when it comes to evaluation paradigms and models.

I shall now womble off and try to find some articles :D

Week four: Evaluation paradigms and models

Must admit I was thankful for Bronwyn's definition of paradigm this week.
Bronwyn said:

"The word paradigm conjures up visions of stiff and starchy academia doesn't it? Well think of it as a style or belief. You will all have heard the words paradigm-shift when people talk about how eLearning is changing how we teach; meaning the change from teacher-centred to student-centred learning."

Previously, I blogged about a 'brain shift'...was I talking about a similar thing?
I gather, in my words, it essentially means 'a model, or pattern'. Therefore "paradigm-shift' means using a different model or pattern...and may require a "brain shift' to do so?

Then in the readings (see link below) there was the word Hermeneutic! I had to look that one up! Grief! (should also add - all good, I enjoy word puzzles)
Seems it means: 'Interpretive; explanatory' (dictionary.com)

According to Professor Tom Reeves there are four Educational Paradigms:

1. Analytic-Empirical-Positivist-Quantititive Paradigm: the most established of the paradigms. My interpretation is this is something that can be measured, such as a set of statistics. Supposedly objective...but I do wonder if the viewpoint of the surveyor has an impact here...
   
2. Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm : In my words - this is the analysis of comments regarding how we (humans) do things. I still don't really understand this but hope I'm on the right track...apparently subjective.
   
3. Critical Theory-Neomarxist-Postmodern-Praxis Paradigm: Reeves suggests proponents of this paradigm are 'social activists' and "view themselves as the forces of liberation engaged in mortal conflict with the powers of liberation". Must admit I'm not too sure about this one and would need to research more before making a comment but my gut feeling is that it sounds far to political for me.
   
4. Eclectic-Mixed methods-Pragmatic Paradigm: I'm still reading and re-reading this. Reeves suggests this is the "most useful" paradigm. And I gather it is a mix of the above...? (source: paradigms by Professor Tom Reeves.)

While I'm still digesting all of the above, I wondering about the impact technology has has on our lives...and how we measure that impact.

I think I need to stop thinking :)

More Week 3 - Readings

The evaluation study was interesting.
I was very surprised to read that the students preferred audio over video!

Note to self:
(from the link above)

"Lessons Learned
The unanticipated results of this project strongly reinforced two lessons that can be applied to most learning technology projects: (1) the importance of actively involving the client, and (2) the importance of using proven instructional design and formative evaluation techniques."

Week 3 - eLearning Guidelines for Quality

Over the past few days I've been enjoying the Easter sunshine and slowly exploring the eLearning Guidelines for New Zealand at http://elg.massey.ac.nz/ What an interesting site!

My task for this week is to:

• Select two elearning guidelines relevant to my area of practice - these will be used in my evaluation project
• Identify two issues related to quality in eLearning - these will provide part of the rationale for my evaluation project
• Post the two eLearning guidelines I have chosen with an explanation of my reasons for choosing them

------

Well...[shuffles feet] the above was all I was able to write (as a draft) over the Easter Break! However during the past week I have been reflecting on what I've read...so, for what it's worth:

Intro and rationale:
We have been using BlackBoard for approximately 6 years - mainly for online-summative-assessments of the theory elements of Level 2, Hairdressing unit standards.
These assessments are conducted under 'exam conditions', on site, in a computer lab.
When we introduced this form of assessment we had excellent feedback from our younger students (ie those <25)>

• will the technology work on the day of the timetabled class assessment?
• do I have a Plan B if the technology doesn't work?
• if a student cannot attend the timetabled assessment time, how do I get them onto a computer and ensure the same 'exam conditions' ?

I think the main thing I have learned from this is ensuring the teaching staff are comfortable with the technology used. If they are comfortable then the students feel comfortable using this form of assessment.

With the above in mind, I have chosen guidelines which I hope will address both my students' and particularly my colleague needs.

Two Guidelines relevant to my area:
1: TD7 Can students practice using the assessment tools used in the summative assessment?
Comment: IMHO our next step is to update our formative assessments to better familiarise my colleagues and our students with the summative assessment.
Issues:
- "Brain Shift". While my colleagues can see the value in online assessment there seems to be some reluctance to produce online formative assessments. Currently we still seem to be 'stuck' in a paper based formative assessment style which (to me) has no bearing on the summative assessment conditions.
The issue here could possibly be related to...(see next issue)
- Professional development of the potential of e-learning, the tools that can be used, and the people who can produce them.

2: TO8 Is subject and instructional design expertise available to staff to develop and support quality e-learning courses?
Comment: I think this is very much related to point 1. Yes it is available...but do we make the most of it? I doubt that we do.
Issues:
- Moving from "chalk and talk" to empowering the learner to have a variety of opportunities to learn (another 'brain shift'?)
- Time...I'm sure we can all relate to this one. Especially time for training. Having, for eg, Templates makes life much easier but staff still need to be trained in how to use them.

Hmmm...reading back on this I'm not sure that I have completed the task for the week 3 - I look forward to your feedback :)

Week 2 - Quality and Evaluation

It's now officially Week Four - I'm still here and still in 'catch up mode'...
During Week Two we were asked to answer three questions about evaluation. I've found these have taken a bit of time to think about and I've realised there's a lot more to 'evaluation' than I had previously considered.

So, on to the questions and my answers:

Why is evaluation important to you and how do you define it?
To get my thoughts in order I had to consult a dictionary or two re 'evaluation"
The definition that best worked for me was found at dictionary.com under 'programming'...

"2. The process of examining a system or system component to determine the extent to which specified properties are present."

Other definitions were around the ability to 'form an idea', the 'value', 'worth' and the ability to 'assess' something. To me these could be subjective...
I prefer the definition above as (to me) it seems more objective - ie: here is the criteria, has it been met or not?

So why is it important to me?
Seems to me that if evaluation is done correctly it would be an efficient tool which would indicate if I am meeting the intended purpose.

What sort of evaluations mentioned on the presentation are familiar to you already and why?
Currently (in our predominately F2F teaching/learning environment) we use both formative and summative evaluation.
To explain:
Formative: Around weeks 3 - 6 we ask the students to evaluate the programme so far (ie. how are they finding the course/programme, do they know where to find support, are they enjoying the content etc). This is done via an online BlackBoard survey which is anonymous and only the collated stats and comments are visible. The results are circulated to the tutors.

Around mid-semester we ask students to evaluate their individual tutors. This asks specifics re tutor teaching methods, feedback etc. This is paper based and is collated by an independent person so once again the information collected is anonymous and the collated stats and comments are sent to the individual tutor.

Around the middle of each term we hold F2F meetings with a student rep. from each class and ask them to put forward suggestions/concerns that their class/group may have. This is summarised as minutes and available to both tutors and students.

Summative evaluation: At the end of each programme we ask students to evaluate the programme via an online Blackboard survey.
We then have a verbal discussion with the group as some prefer to talk in a group atmosphere rather than writing their thoughts down...both of the above are circulated to tutors.

Finally, statistics on course course completions etc. are summarised and explained/commented on in an Annual Programme Report(APR). The APR is circulated to all staff within the department, agreed upon and then sent to the Academic Development Centre (ADC) for comment/review.

The last two summative evaluations I also consider to be formative - ie forming an idea of what should change for the next intake of students.

Why is quality important in eLearning?
Personally I find this q. difficult to answer. To me eLearning is just another form of teaching/learning. There needs to be quality in both F2F and eLearning.

I wonder if I am being naive with this thought?

Flight of the Conchords

Part of the presentation we needed to view in week two (see below) was a You Tube clip by Flight of the Conchords - See: Bowie Song.
Personally, I think these guys are great and I can't understand why NZ and TVNZ didn't embrace them...but that's my own viewpoint.

The questions asked within the presentation, along with my answers, follow:
1: Is this good quality?
Seems OK to me. I've seen better resolution and such on You Tube...but the message and humour still gets across - also a quick download (is that what you mean by 'quality'? )

2: Why is it good?
I enjoy their humour and I think their musical presentation is great!

3: Why is it not so good?
Must admit, I'm left asking questions here:
- Who else shares my sense of humour?
- Does humour relate across cultures?
- Does humour relate across age groups?
- Should I impose my sense of humour on others whom I don't *really* know? (ie my students)

Call me old fashioned..but when it comes to humour I think it needs to be relevant to the students and the learning context.

Quality and Evaluation - Week 2

The first task for Week 2 was to watch a 20 minute slide presentation Why is evaluation so important . I found this slide show interesting - for 3 reasons:

Firstly: I have heard of myplick before but have never seen it used, so I needed to check this out. It seems it's free (always good when on a tight educational budget! ) and apparently one is able to share/embed presentations (including audio) on the web.
This sounds like something worth trying. (see myplick FAQ for more info )

Secondly: I discovered something about my own learning - a 20min presentation was too much for me! While the presentation was very informative I found myself twiddling my thumbs and looking out the window. I couldn't concentrate for that long with out doing something.
I found myself watching it in 3 stages over a couple of days...I guess that would be about 6 minutes each? Then, I was able to grasp some of the info ...and I think I'm finally starting to digest the information and understand it after reviewing it for the fifth, or sixth time.

Thirdly: Must say it has been nice to be able to go back and review. This is not something that can easily be done in a F2F presentation.

Key points of the presentation:

Evaluation - is it really necessary

What is evaluation?

What are you evaluating - context is important

Why evaluate

How to evaluate

Evaluation methods

Designing an evaluation plan

Evaluation reports

Resource

It seems there's a lot to consider when it comes to evaluation.

No Pressure...Yeah right! :)

Pressure...can it motivate or de-motivate a student?

Surprisingly I just realised I fall into the former category (well, at least for today!)
I've just started my 5th paper in the Graduate Certificate in E-Learning...unfortunately I was two, nigh on three weeks late in enrolling ...right now I'm wondering if I can catch up.

I think I can...this is something I'm determined to do...but I think there are a few things to take into account here:

#1 There has to be internal motivation. Today I want to achieve...will I feel the same tomorrow? Right now, I'm sure I can but who knows what challenges tomorrow may bring for the keen student.

# 2 (From above) Who will inspire me tomorrow? My online teacher? My online colleagues? Me?

#3 Is the pressure achievable? For me, today, I think so....

But I am left wondering...
How does one phrase an encouraging message to student who is late to enrol?

Sue