I hope this will help you guys. I am a statistician by trade, so here are my suggestion on your scoring system.
1. It would be nice if you can add clear instructions on how folks should score. Give value labels for a few other points on the scale, and not just the two anchor points. You'll be surprise how differently folks interpret the rating scale you have given.
Personally, when I first look at a submission, I start at a neutral score 5.5. Then if I feel the photo meets the minimum requirement, I move up a notch to 6. If it doesn't meet the minimum requirement, then I knock it down to 5. If it has a little extra umpth (in terms of content and/or composition), I move up to a 7 or 8. If it has a lot of umpth, I move up to a 9 or 10. 10 is reserved for photos that I would be really proud to have taken it myself (although, I rarely believe my own shots are above 8's). Overall, you can tell my scoring system is biased toward grading how good a submission is, and not neccesarily how bad it is. Although, I do give scores 5 and below, they are far an few in between. Now, I don't know how many others follow my system. But I wouldn't be surprise if there are four to five other ways that folks approach your scoring system (10-point scale with 2 poles (bad vs. good)).
2. The description of my positive bias when scoring on a bi-polar scale alludes to my second suggestion. Make it a uni-polar scale, and drop the 'bad' pole. That is, simply use a scale based on 'good' (1=not very good, 2=somewhat good, 3=moderately good, 4=good, 5=very good) or (1=not very good, 3=somewhat good, 5=moderately good, 7=good, 9=very good). You may wish to give specific elements that people should be looking for at each score level.
Reason for not using bi-polar scales: 1) People tend to be biased toward one end of the scale, 2) The overall score is based on the average of all scores given, which requires that differences in scale values have sensible meanings. On the current (bad vs. good) scale, it is easy to interpret a difference between 6 and 8; but a difference between 1 and 9 is harder to interpret if at all possible. In order to interpret this, you need to mentally convert them to the same thing (e.g. a measure of goodness). Now as yourself this, is 'very bad' equivalent to 'not very good'? Probably not. Averages just do not work with bi-polar scales.
Well enough of my driveling. I hope this insight into likert scaling is useful to you guys, and anyone else who is interested. I bet the next time you all look at a survey, you won't see it the same way you used to.
* This message has been edited by the author on 3/25/2002 4:14:46 PM.
* This message has been edited by the author on 3/25/2002 5:02:49 PM. |
