| Author | Thread |
|
|
10/09/2002 04:05:00 PM · #51 |
The problem is that every person have a different idea of the scale in the 1-10 or 1-7. I'll bet tehre are a lot of people that doesn't like the photo and gives a 1 or some that gives a 5 for the lowest mark they'll give sothat they don't offedn the other guy. Both of them skewed the average scores which is used for ranking. Not everyone distributes their votes across 1-10 scale the same way and the results are meaningless except for the 8-10 votes which tends to be "good" votes. So I am proposing, let's count 8-10 votes and see what it shows.
Originally posted by iggy386:
Originally posted by paganini:
[i]That's my educated guess. I think it's fair to say that the top 3 photos on here probably doesn't deserve a 1. You can say that it's average due to your own comparison, that should get a 5-6. But yet we see 1-4 scores for the top 3 every single time, which suggest that some are voting based on personal taste.
Yes, people vote by personal taste and their opinion. You make it sound like that's a bad thing. If it was a science, all we would need is a to let a couple of people who know what they're doing look at each photo and rank them. There are some generally accepted guidelines as to what works best and what doesn't, but photography is not a science.
The fact that some people always give low scores to the top rated photos (and some give high scores to the lowest rated photos although nobody ever complains about that) has been discussed on here repeatedly. So what? Since anyone can vote some flukey votes are always going to be cast. It's unavoidable and totally unimportant. It does not invalidate a voting method that lets people judge photos on degrees of goodness rather than just ok or not ok.
So even if people are voting on whether or not they like a photo, as opposed to whether or not some mythical expert would objectively judge the photo as good, I don't see the problem. The 1-10 or 1-7 voting scale allows people to say that photo X is better than photo Y and a heck of a lot better than photo Z. That works for me.
Mark
[/i]
|
|
|
|
10/09/2002 04:08:40 PM · #52 |
Paganini, what's the purpose in just counting the 8-10 votes?
|
|
|
|
10/09/2002 04:25:51 PM · #53 |
Attempt to normalize the voting. :) That way, we don't have to worry about how theuser votes on a 1-10 range (i.e. one person's concept of a 1 is different from another person's 1 vote, some use it for anything they don't like, other don't, etc). The common ground is that 8-10 seems to be what the user considers as a good image or what they like.
I suspect if an image doesn't do anything for the user as in "This photo doesn't do anything for me" :), it will not get a 8-10. So you count the number of votes in the 8-10 range, divided by the number of votes you have received, and that's a percentage number you can use for ranking. Problem with the AVERAGE is that the one user's concept of a 1 score versus the other user is NOT the same, but they seem to have the same idea when they vote the highest scores. The real hazy area is the 5-6 voting. To me, a 5-6 means a 1, because unless the viewer likes it or gets the message, the photograph isn't effective.
Originally posted by JohnSetzler :
Paganini, what's the purpose in just counting the 8-10 votes?
* This message has been edited by the author on 10/9/2002 4:26:09 PM.
|
|
|
|
10/09/2002 04:57:20 PM · #54 |
John: I really like the idea of a guideline such as you have outlined above. I think this would aide in the "fine tuning" of people's thoughts. Perhaps it would also diminish some of the issues that paganini is voicing, and that we have all felt at one time or another.
Great idea.
|
|
|
|
10/09/2002 05:02:00 PM · #55 |
John,
As reasonable as those suggested ratings are (very similar to what I use already) I believe if you try and define an "official" DPC voting scale you will initiate the Thread of Ten Thousand Posts...
What about my idea of having to rank all the photos? |
|
|
|
10/09/2002 05:07:49 PM · #56 |
generale, how would you address the many people that have time to vote on a significant chunk of the photos, but not all of them? somehow those votes should be taken into account, too, or i suspect we'd have a much smaller number of votes per pic to base the final ranking on.
|
|
|
|
10/09/2002 05:14:16 PM · #57 |
Pag. I am math blind so I am no following your argument. Could we get an example please?
Your relection picture ranked 100. 30 people did not like it. 32 liked it. 218 by your recomendation don't count??? or have a neutral opinion.
My picture by comparison ranked 134. 35 did not like it. 30 did. 220 scores fell between 4-7.
aelith -- curious and puzzled |
|
|
|
10/09/2002 05:24:17 PM · #58 |
Unvoted photos could be assigned the average of the remaining rankings.
I can't do the math right now, but something like this:
Total submissions: 100
Ranked: 50
Unranked:50 assigned score X where
X = (51+52+53...+97+98+99+100)/50
In a ranking system, lowest score wins...
Hardest part is arranging for everyone to see the large images before dragging the thumbnails. Maybe a combination would work: assign a numerical score as presenly done, but rank photos within scores on the vote summary page. That would have somewhat the effect of allowing folks to assign a score of, say, 4.8 for "I wanted to, but couldn't give you a 5" by ranking it as the top of the 4's, and have the effect of weighting the votes instead of applying a strct arithmetic mean. |
|
|
|
10/09/2002 05:24:22 PM · #59 |
Have the admins considered adjusting individual votes based on the voter's overall average? Obviously, not everyone votes on every image, though alot of us try. I'll bet that this will be the case more in the future as the submission numbers keep going up. Consider the following: two voters only vote on half the photos each week, one averages a 4 across all challenges, the other a 7. The problem I see is that it is hard to rank photos that have been judged by different sets of voters.
Chess players might know what I'm talking about. Look at some of the formulas used to rate players. The idea there is the same, be able to accurately rate players that have not necessarily played the same set of opponents.
There are all kinds of funky statistical formulas, curve fitting, and probabilistic mumbo-jumbo you could do to adjust a voter's votes so that they are true to the site average. Have the admins considered trying some sort of advanced analysis of the votes, just to see how it affects the results?
My guess is that switching to a narrower scale might correct this problem on its own. A 1-5 scale would force voers to work more closely to a certain average, and as long as there are enough votes ties shouldn't be a big problem. To see what I mean, imagine the scale being 1-100. I think people's vote averages and distribution curves would be all over the place. |
|
|
|
10/09/2002 05:35:10 PM · #60 |
One "problem" with all of this is that improving the "accuracy" of the score really emphasizes the competitive aspect of the site, while we are finding the exchange of ideas and techniques suffering.
I would love to get a ribbon, but I am going to submit the photo I want people to see, not necessarily the one I think will get the highest rating here, so a more "accurate" score is relatively unimportant to me. And, I think the randomness of the current system (or lack thereof) provides a pretty accurate result anyway, especially given how relatively few outraged "why wasn't this a top 3" situations we have... |
|
|
|
10/09/2002 05:49:48 PM · #61 |
| I think that we should have the 1-10 scale, but I also think there should be a scale where we vote on the pictures problems. Like focus, lighting, contrast, and so on. I think with this we would beable to see what is wrong with our photo and not have to worry about people giving us comments. It would also eliminate the people who vote on whether they like it or not. Do you think this would work? |
|
|
|
10/09/2002 06:27:14 PM · #62 |
Originally posted by Sonifo:
It would also eliminate the people who vote on whether they like it or not. Do you think this would work?
Why should they be eliminated? Surely, whether one likes something or other is actually more important than the technical quality. A lot of people really like "The Osbournes" TV show, for example, even though the quality of the production is very low compared to, say, "The West Wing". To say that those people are somehow "wrong" is elitist in the extreme.
|
|
|
|
10/09/2002 06:33:28 PM · #63 |
Nah, if 32 liked it, then it's 32 / (218 + 30) is the ratio that you use for my ranking, so taht's 12.9%. Then you get a percentage of people who liked it.
And in your example, you'd get 30 / (35 + 220), which is 11.7%
That's not as much important as that I'd want to know who voted that they liked it versus who doesn't :) that's more important than getting an average score or even the ratio of people who liked it.
Originally posted by aelith:
Pag. I am math blind so I am no following your argument. Could we get an example please?
Your relection picture ranked 100. 30 people did not like it. 32 liked it. 218 by your recomendation don't count??? or have a neutral opinion.
My picture by comparison ranked 134. 35 did not like it. 30 did. 220 scores fell between 4-7.
aelith -- curious and puzzled
* This message has been edited by the author on 10/9/2002 6:34:43 PM.
|
|
|
|
10/09/2002 06:44:42 PM · #64 |
Those are technical merits and wouldn't account for the score.
FOr example, Journey had a great reflection entry ( at least i thought it was awesome ) where it was slightly out of focus, making the bottle look like a painting. It was voted down for "being out of focus", when in fact, if it was in focus, it wouldn't be interesting. If all we care for is simple technical merits, then the personwith the clearest photo will win each week, even the images that is devoid of feelings and emotions.
At the same time, the other image that got a very high rating from me was the #1 winner where i voted it based on how I felt when I looked at it. If it was taken on a clear day and without the mist which caused it to be slightly out of focus, i'd probably rated it down lower to a medium level photo as i did with most water reflections.
Then again, i tend to like photographs that have emotional content.... or 'artistic' if you want to use that word. Unless the competition was photo-journalistic style, a snap shot photograph would get a low score from me. Again, i am applying my personal preference here.
Originally posted by Sonifo:
I think that we should have the 1-10 scale, but I also think there should be a scale where we vote on the pictures problems. Like focus, lighting, contrast, and so on. I think with this we would beable to see what is wrong with our photo and not have to worry about people giving us comments. It would also eliminate the people who vote on whether they like it or not. Do you think this would work?
|
|
|
|
10/09/2002 08:51:59 PM · #65 |
This thread is starting to sound more and more like a Florida election board - "how do we set up the voting system so that the voters vote the way we want them to instead of the way they want to?"
Guys - there's nothing wrong with the voting/scoring system now. Let's face it - there is no single standard for what makes a good photo. If voters don't like your photo, why manipulate the stats to make it look like they do? If they wouldn't buy it, they'll vote it lower than one they would buy. The scores between 3 and 8 allow the rest of us to have some differentiation from the really good and the really bad.
Part of the magic of DP Challenge is that nobody has set rules for how we are supposed to vote. Part of the statistical beauty is the widespread geographical, cultural, racial, and historical background that speckles the nature of the site. Part of the reason it's hard to please the voters is because not all the voters think alike. Enjoy it. Let's stop trying to make them all vote the same way. Why are there multiple judges at a skating contest? Diversity is key to successful population sampling. Let's don't fight it. Let's enjoy it.
|
|
|
|
10/09/2002 09:10:23 PM · #66 |
| Ok so we should not eliminate the voters that score for how much they like it, But I would still like to see the other way to. Maybe if we could get them both scored together. I do vote for what I like, but if I see a photo that I don't like I will vote it on how well they took the photo and just not give it a 9 or a 10. I think everyone has a good point. Can we have it all? lol |
|
|
|
10/09/2002 10:03:31 PM · #67 |
Paganini has a valid point with the dilution of scores caused by some people voting 1-4 for below average pictures and others voting 5 or 4-6 for below average pictures. I don't believe this could be remedied with the rating system guidelines suggested by Setzler. I have trouble differentiating between his 2, 3, and 4. You would almost prefer to get a 2 because that one shows some promise whereas 3 and 4 don't have that hope of ray. If a picture has serious flaws how are we to be able to tell whether there is any promise on the horizon? And yeah, anyone can improve with "some hard work". You could add that caveat to the picture you give a 10 as well :)
How are you supposed to rate a brightly colored, in focus snapshot? What if you knew the photographer has been submitting brightly colored, in focus snapshots week after week for the last 25 challenges? You don't know that, of course, when voting, but then it would really be time to say "Hey, look, your work is seriously flawed. If you want to keep on doing just snapshots, power to you. But if you are serious about photography, start learning more about photography and put in some extra effort". The reality is that brightly colored, in focus snapshots, accompanied by a cute title, are not doing so badly at dpc. According to John's rating system, some people would give the snapshot a 1, because they perceive a snapshot as "seriously flawed" for a photography site, others would give it a 5 and consider it an average photo. In other words, the dilution would continue.
Again, I'm more interested in who is giving what vote to whom. Make everyone's vote public. More informative, and you can figure out what voting system voters have adopted.
Originally posted by JohnSetzler :
I agree completely... I think that a guideline associated with each number on the 1-10 scale would be very useful. Maybe we could start a suggestion list:
1 = Extremely flawed work.. needs serious improvement
2 = Many flaws but shows some promise with some hard work
3 = Below Average
4 = Slightly below average
5 = Average Photograph
6 = Slightly above average, but needs some additional work
7 = Good photograph
8 = Good photograph with some nice extra elements
9 = Very nice photo, but not quite perfect
10 = Excellent Photo - Great Work
Something along these lines being associated with the scores maybe?
|
|
|
|
10/09/2002 10:26:22 PM · #68 |
If there are no rules to follow, then all we have is chaotic voting :) which explains what it is.
There really isn't any difference between someone who gets a 50th place versus someone who gets 100th place, statistically they're the same because the average doesn't shift much at all.
And if you really think about it, the difference between 5 or 6 or 7 score is not much at all, but in the rankings it makes a huge difference for a 0.4 difference in the average, it just means someone get more votes of 6 than the other person who gets more of 5, and 5 and 6 have no meaning at all. A difference between a vote that says "I like your photo" and one that says "I don't like your photo" is much clearer, at least you know where the person stands. I for one considers any votes i get from 1-6 as "no i don't like your photo or i am indifferent".
Originally posted by nards656:
This thread is starting to sound more and more like a Florida election board - "how do we set up the voting system so that the voters vote the way we want them to instead of the way they want to?"
Guys - there's nothing wrong with the voting/scoring system now. Let's face it - there is no single standard for what makes a good photo. If voters don't like your photo, why manipulate the stats to make it look like they do? If they wouldn't buy it, they'll vote it lower than one they would buy. The scores between 3 and 8 allow the rest of us to have some differentiation from the really good and the really bad.
Part of the magic of DP Challenge is that nobody has set rules for how we are supposed to vote. Part of the statistical beauty is the widespread geographical, cultural, racial, and historical background that speckles the nature of the site. Part of the reason it's hard to please the voters is because not all the voters think alike. Enjoy it. Let's stop trying to make them all vote the same way. Why are there multiple judges at a skating contest? Diversity is key to successful population sampling. Let's don't fight it. Let's enjoy it.
|
|
|
|
10/09/2002 10:36:32 PM · #69 |
well said. I also like to know who voted good or bad toward my photo. If you know their work online, then you also know how they vote, that has more of a meaning than getting average scores.
And yeah, i am the type that would give a bad score for a snapshot photo that i find boring or devoid of meaning. But that doesn't mean a well executed photo is bad. In the reflections i probably gave low scores for any water reflections that aren't dramatic or just a simple touristy snapshot, and all of them have made it to the top 50. The only good water reflection i gave was the first one because it conveyed a sense of beauty and enchantment that the others failed to deliver. In fact the guy could have blurred the background it would've still made a huge impact.
And what's up with the FLOWERS???? I find every challenge at least 3-4 studio shots of flowers. Alright, maybe i should add something like that to my totally disturbing SIN submission to be done this weekend, maybe i'll get a higher score by diluting my message with a flower :)
(I'll just have to add a teddy bear or a puppy to get that extra 0.5 raise in the average....)
Originally posted by Journey:
Paganini has a valid point with the dilution of scores caused by some people voting 1-4 for below average pictures and others voting 5 or 4-6 for below average pictures. I don't believe this could be remedied with the rating system guidelines suggested by Setzler. I have trouble differentiating between his 2, 3, and 4. You would almost prefer to get a 2 because that one shows some promise whereas 3 and 4 don't have that hope of ray. If a picture has serious flaws how are we to be able to tell whether there is any promise on the horizon? And yeah, anyone can improve with "some hard work". You could add that caveat to the picture you give a 10 as well :)
How are you supposed to rate a brightly colored, in focus snapshot? What if you knew the photographer has been submitting brightly colored, in focus snapshots week after week for the last 25 challenges? You don't know that, of course, when voting, but then it would really be time to say "Hey, look, your work is seriously flawed. If you want to keep on doing just snapshots, power to you. But if you are serious about photography, start learning more about photography and put in some extra effort". The reality is that brightly colored, in focus snapshots, accompanied by a cute title, are not doing so badly at dpc. According to John's rating system, some people would give the snapshot a 1, because they perceive a snapshot as "seriously flawed" for a photography site, others would give it a 5 and consider it an average photo. In other words, the dilution would continue.
Again, I'm more interested in who is giving what vote to whom. Make everyone's vote public. More informative, and you can figure out what voting system voters have adopted.
Originally posted by JohnSetzler :
[i]I agree completely... I think that a guideline associated with each number on the 1-10 scale would be very useful. Maybe we could start a suggestion list:
1 = Extremely flawed work.. needs serious improvement
2 = Many flaws but shows some promise with some hard work
3 = Below Average
4 = Slightly below average
5 = Average Photograph
6 = Slightly above average, but needs some additional work
7 = Good photograph
8 = Good photograph with some nice extra elements
9 = Very nice photo, but not quite perfect
10 = Excellent Photo - Great Work
Something along these lines being associated with the scores maybe?
[/i]
|
|
|
|
10/10/2002 12:09:32 AM · #70 |
edited out my original post - I just saw that John Setzler already proposed a refined scoring system, much better than I did:
1 = Extremely flawed work.. needs serious improvement
2 = Many flaws but shows some promise with some hard work
3 = Below Average
4 = Slightly below average
5 = Average Photograph
6 = Slightly above average, but needs some additional work
7 = Good photograph
8 = Good photograph with some nice extra elements
9 = Very nice photo, but not quite perfect
10 = Excellent Photo - Great Work
I think that a scoring system like this will help to level the voting playing field (like the thumbs up/down method at www.photosig.com.
* This message has been edited by the author on 10/10/2002 12:09:37 AM.
* This message has been edited by the author on 10/10/2002 12:19:50 AM. |
|
|
|
10/10/2002 12:46:01 AM · #71 |
I don't agree with your levels of refinement. Nothing short of a 50 point scale, devided into 10 point catagories, will do justice to the work that is presented here. Can't handle it? Don't vote. The "subjective" factor should be defined, require comment, and limmited to a yes or no vote, equal to 10 points.
* This message has been edited by the author on 10/10/2002 12:50:30 AM. |
|
|
|
10/10/2002 03:40:47 AM · #72 |
Originally posted by rapsiii3:
I don't agree with your levels of refinement. Nothing short of a 50 point scale, devided into 10 point catagories, will do justice to the work that is presented here. Can't handle it? Don't vote. The "subjective" factor should be defined, require comment, and limmited to a yes or no vote, equal to 10 points
I disagree. For subjectivity, I could not vote YES/NO because I definitely believe that some subjects are better than others. I don't believe that subjectivity rides on a yes/no vote from my point of view. I'm not cut and dried like that. I rate subjectivity on the same scale as everyting else... poor, below average, average, above average and excellent... How would I go to a yes/no on something like that?
|
|
|
|
10/10/2002 03:43:19 AM · #73 |
I would also love to see a multi-level vote where there was a 10 point scale on categories such as:
subjectivity
technical excellence
creativity
artistic merit
composition
etc...
Scoring like this automatically provides some amount of feedback to the photographer. Even without a comment, you can get a feel for how the voter views your image...
|
|
|
|
10/10/2002 03:49:51 AM · #74 |
Originally posted by JohnSetzler :
Originally posted by rapsiii3:
[i]I don't agree with your levels of refinement. Nothing short of a 50 point scale, devided into 10 point catagories, will do justice to the work that is presented here. Can't handle it? Don't vote. The "subjective" factor should be defined, require comment, and limmited to a yes or no vote, equal to 10 points
I disagree. For subjectivity, I could not vote YES/NO because I definitely believe that some subjects are better than others. I don't believe that subjectivity rides on a yes/no vote from my point of view. I'm not cut and dried like that. I rate subjectivity on the same scale as everyting else... poor, below average, average, above average and excellent... How would I go to a yes/no on something like that?
[/i]
Then why allow subjectivity in the scale at all? You like it or you don't. Subjectivity is the least dependable measure of quality we have, yet it is important. A simple yes or no, equal in measure, but qualified by additional votes on technical matters would be more helpful.
|
|
|
|
10/10/2002 04:15:00 AM · #75 |
In order to follow this method, I would have to say YES, I like it subjectively, or NO, I don't?
That's absurd...
|
|
