I responded to Matt Thomson’s evaluation plan on Blackboard, but in doing so found out a few things that I will now consider for my own evaluation. This has been a good exercise for that reason. (p.s. Matt’s original evaluation plan isn’t on his blog yet, but hopefully it will be soon.)
Below is my response.
I thought I would respond to your post – I see that it is similar in many ways to mine, so it’s quite difficult to look at in this way… I thought I would focus on your points of difference, and in this way, expand my own learning.
You discuss using triangulation as a methodology for collecting and analysing data. In his book Doing Qualitative Research, David Silverman makes a concise and clear analysis of the pros and cons of doing so. He argues that many qualitative researchers believe that triangulation provides more reliable results than any single method, but warns against some of its pitfalls. He suggests that its success may depend on your analytical framework (Silverman, 2005) and should you choose to work in this way you set out the following, clear guidelines:
- “Always begin from a theoretical perspective or model
- Choose methods and data which will give you an account of structure and meaning from within that perspective (e.g. by showing the structured contexts of the interactions studied.)” (Silverman, 2005)
He surmises the chapter with a preference for the single method:
“It is usually far better to celebrate the partiality of your data and delight in the particular phenomena that it allows you to inspect (hopefully in detail).”
I feel that because in this instance, you would be analysing different parts of your overall design and working with different groups of people that triangulation may be the most suitable choice of methodology, as long as you have in mind the ways in which you might analyse the diverse data it yields during the evaluation stage.
Later on you discuss what methods you would use to collect your data, and similar to my own evaluation you have turned to the some of the OP methods already in play, and that are readily available to us, such as BB, and paper questionnaire forms. I wonder, if in light of the complex nature of triangulation, that there is some way of streamlinging these methods so that they are easily referenced, or as Silverman suggests, you agree on a set framework for analysis before deploying any of them.
I also wondered about taping, visually, or aurally, any focus groups you might get together in order to keep data for analysis. I believe that this is problematic in itself, as people can feel uneasy faced with such permanent technologies, adn don’t feel as free to speak. However, a note take may be impartial, framing what is recorded to favour one outcome or another, (without intention!) and therefore not allowing your data to be left open to different types of interpretation, in light of some of your other results.
One thing that occurs to me, is that I find it can be difficult to get students to fill in (endless) questionnaire forms. This of course depends on many things: the individuals in the class; the time of year; the way you ‘sell’ it to them. I have been looking at Learning Contracts as a potential method to work with some of our students who can often feel a little disenfranchised between year 2 and year 4 of the BFA. This is a transitional year, where the students are encouraged to be self-directed with the skills they have so far learnt, developing both their practice and their own ability to co-ordinate that development. In Using Learning Contracts in Higher Education, Laycock and Stephenson state that “there is much evidence from student, teacher and employer testimony that involving students in the design of their own programmes of study improves the quality of their learning, increases motivation, promotes understanding of fundamentals and focuses student attention on the wider relevance of their studies.” (Laycock and Stephenson, 1993) I propose here that this could be a possible methodology for structuring focus groups that involces the stakeholders that you mention above.
Once again, I hope that this provides you with some more information that will help with your evaluation process. I think that your evaluation plan is strong, reliable and well-rounded, and importantly has a good balance of investment/reward. Good luck with the course!
Silverman, D. Doing Qualitative Research Sage Publications, London (2005)
Laycock, M., and Stephenson, J. Using Learning Contracts in Higher Education Kogan Page, London (1993)