*** A New Voting Scheme *** - DPChallenge Forums

Canon EF 28-135mm F/3.5-5.6 IS USM

07/09/2004 04:41:28 PM · #1

Please read revised entry on this idea on the 12:38 entry in this thread.

I am new here, but after following some threads, I find the sight very attractive and useful with the exception of the current voting system.
The fact that each voter with an entry has a vested interest creates a schism of voting styles. These systems are created not necessarily to win a ribbon, but to maintain a high average vote received. It is this motive that makes many pranksters give ones and twos to otherwise good images. This brings the loser images closer to the winners.
The way around this is to split the voting into two equal groups, let us called them group a and group b. Group a votes on group b and vice versa. Vested interest is gone. Any score or vote you give has no effect at all on one's own image which is in another voting block.
The only draw back is that the votes will be 50 percent less per challenge. Members can receive either group a or b as determined by random.
The second draw back is the similarity of the combined results. Yet, this may not be as serious. For example, ties for ribbons are broken by the site council and an algorithm can handle the lower strata ofoccasional duplication.
Anyway, this scheme needs to examined by mathematician.

If this idea is too radical or cumbersome to implement I would suggest that the current software should analyse each stats on a given picture and if the the average is above say 5.2 or whatever figure to discard the ones and the 10's. Yes, voting is personal and subjective, but even a bad idea well exposed and decently composed that has received an average of 5.5 does not deserve a one nor a ten. As a note: I have submitted the wrong files to two challenges. This has cost me dearly and they deserved the low votes they received, however, this is not about me. I have not been here that long to complain, but the general disatisfaction I see in the threads is alarming. To me the problem boils down to the vested interest.

Message edited by author 2004-08-03 01:16:54.

07/09/2004 04:45:07 PM · #2

This idea has been discussed before (big surprise, right?)

Personally, rather than seeing the voting split into 2 groups, I'd just rather have a challenge entrant's scores/comments hidden until they have finished voting (by clicking an "I've finished voting" button, which would become active after voting on 25% of the images), so as not to be biased (even if subconsciously) by their own score when voting on other entrant's photos. Of course you would still be free to go back and comment on entries after you've "locked" your votes.

While not exactly the same as what you proposed, I like being able to see and vote on all the pictures.

Your other suggestion sounds somewhat similar to the concept of "voting karma" that I described in this thread and this thread.

Message edited by author 2004-07-09 17:04:07.

graphicfunk

Canon EF 28-135mm F/3.5-5.6 IS USM

07/09/2004 05:06:42 PM · #3

Hey EddyG: I did follow those threads. However, the vested interest persist. I think the real way out of the dilemma is for entrants on one challenge to vote only on another challenge. This even elimates all the complications and draw backs of the first suggestion. Again, just more food for thought. There must be a way to improve this.

Canon PowerShot Pro 90 IS

07/09/2004 05:16:25 PM · #4

There are really two issues here, the vested interest and "troll voting", e.g. someone voting all ones.
There already is an algorithm that looks for suspicious voting patterns. Votes can and will be thrown out if someone is troll voting. There are really very few of these votes, and this can be shown by statistical analysis of the voting patterns (I've personally done it).
It probably would not make sense to change the voting scheme just to deal with a very few outlier votes.
Regarding not voting on a challenge you've entered, that would cut down voting drastically, since many enter 50% or more of the available challenges.
While I agree that a vested interest exists, my own personal opinion is that the effect is very small.

wwwavenger

07/09/2004 05:17:42 PM · #5

Karma systems have shown to be very effective. I would wholeheartedly support such a feature.

A users karma could be based on comments rated by recipients, voting style, length of time spent on the site, participation in voting and challenges.

Message edited by author 2004-07-09 17:18:27.

coolhar

07/09/2004 06:16:08 PM · #6

I'm not real familiar with the karma voting systems but I would be against any change that tends to reward voters for voting similiarly to the masses. Please explain what you mean by karma systems and why they are more "effective" than what we have now.

07/09/2004 06:52:39 PM · #7

Originally posted by kirbic:

There already is an algorithm that looks for suspicious voting patterns. Votes can and will be thrown out if someone is troll voting.

Unfortunately this algorithm is not all that complicated. Drew posted the details of it a long time ago in this thread.

The nice thing about a karma-based system is that it is self-regulating; there are no "hard limits" that have to be coded into some algorithm to determine if a user's votes are "suspicious". It is all based on the image's final ranking.

Message edited by author 2004-07-09 19:14:00.

07/09/2004 07:07:32 PM · #8

Perhaps I don't fully understand how a karma system is supposed to work, but my interpretation is that a voter's karma is in fact based on how well her/his votes are in tune with the masses. I don't think we should be limiting voting in that manner.
Another problem I see is that we really have no way to "reward" good karma. We'd be stick with a negative incentive, i.e. punishing bad karma. Not sure how that would be made to work.

graphicfunk

Canon EF 28-135mm F/3.5-5.6 IS USM

07/09/2004 07:19:45 PM · #9

Here is a voting schemes that I have detected by examining entries in challanges. The spoilers knock the strongest images with ones, twos and threes. They feel this gives their pics a better chance. Now they go and vote high on images which they know will not make it to first base. And this keeps their voting averages up to par.

Obvious also are that vote ultra low but these are consistent and you can read it in their stat: they receive much higher then what they give.
At the risk of sounding redundant, I still believe that voting on other challenges where the voter has no vested interest is the best. You say that this will radically cut down on votes and my reply is:
Create a volunteer voting block and offer an incentive icon to members who commit to voting on all images. You know, this partial voting leaves a lot to be desired. We are here because we like to see pictures and if members are too lazy to want to see pictures then give them an incentive to vote.

Message edited by author 2004-07-09 19:31:05.

Voigtlander Ultron 21mm f1.8

07/09/2004 07:20:02 PM · #10

Your interpretation is basically correct. There is a "karma window" that is +/- 3 of the image's final score. As long as you voted somewhere in this window, you are considered to be voting "fairly" and your karma increases, up to some maximum value. If you vote outside of this window, your karma is reduced (the higher the deviation, the bigger the karma reduction). But remember that this is on a per-image basis. So even if you gave some image a "9" that only rated a "5.5" (meaning the positive-reward karma window would be "2.5 to 8.5"), your karma might be reduced slightly because of that one image, but would increase because of all the other images that you voted inside the karma window. The only people whose karma would be negatively effected are those who consistently vote high-quality images as 1's and 2's.

The weight of your vote to an image's final score is based on your current karma value. So those with "low karma" still have a vote that counts, it just isn't weighted as highly as somebody with "good karma".

Good karma is also awarded by voting on more entries in a challenge. Someone who votes on 100% of the entries will earn more karma than someone who only votes on 20%. Similarly, karma could be awarded for commenting during a challenge.

Worth100.com uses this system, and as they say:

"this is not a system of punishment and reward, nor is it a judgement of your personal tastes. It's merely a way to measure your ability to discern actual image quality, and allows plenty of room for your individual tastes to come into play without damaging your karma. It maximizes the effects that consistantly fair and observant voters have on an entry's average, and minimizes the effects that 'quirky' or downright dishonest voters can have."

Message edited by author 2004-07-09 19:24:33.

MAK

Leica M Monochrom

07/09/2004 07:25:05 PM · #11

I personnally feel (being a reciever of a few 1,2&3 votes) that if a vote of 3 and below was accompanied by a comment as to why the image was given a low score (it also names the person who voted)it my help to filter out the "TROLL" voters simply because I doubt they have the intellegence to make a useful and constructive comment. that way, any vote of 1-2-3 without a reason would be deemed as "TROLL" voting, I know its 'long winded' but it really isn't fair to some people who's images are obviously not deserving of a '1' or '2' (cant think of any examples right now but I see even some ribbons have at least 1 silly vote attached). It does need something to happen to 'weed' out the trollism that is done by people that, to me, seem to just be doing this to harm peoples feelings.

Harryg

Canon EOS-300D Rebel

07/09/2004 07:37:48 PM · #12

I am fairly new to the site and i have noticed this too, also on a few other sites too. at the end of the day everyone is entitled to their opinion be it right or wrong. the few unsportsman like people who vote to improve their own standing are in the minority and i would have thought wouldnt affect the overall outcome, they only end up cheating themselves and achieve nothing in the long run. I do understand what you are saying but at the end of the day if vote censoring takes place then why bother at all, have a panel who votes and leave the comments to the participants.

07/09/2004 07:40:54 PM · #13

Marac: I don't think that would be effective; people don't like being forced to leave comments, and will either enter "x" into the comment field to give you your 1, 2 or 3, or they will simply vote a 4 just to avoid the comment.

For some reason, I think people assume that a karma system won't let people use the full 1-10 voting scale. That isn't the case at all!

Let's say your karma is maxed out at 100, meaning that the weight of your vote is 100% when computing a photo's score. You rate fully 50% of the challenge entries outside of the "karma window" (which would be quite difficult to do if you were trying to vote fairly), giving 10's to images that others thought weren't deserving of it, and 1's to other photos that the rest of the user base decided deserved much higher. Because the other 50% of your votes were within the karma window, your karma adjustment for that challenge would be zero, meaning that the next challenge, all of your votes would still count 100%!

But if somebody purposefully went through and gave 1's to every challenge entry, their karma would quickly fall, and the impact of their votes would be less and less the more they continued to vote unfairly.

A +/-3 karma window is quite huge. That means for a photo that ends with an average of 5, you could rate that photo anywhere from 2 to 8 and be rewarded with "good karma" for voting "fairly" for that photo. Likewise, a good photo, that finishes at 7, has a "karma window" of 4 to 10. Anybody who voted less than a 4 would have a karma reduction for voting unfairly on that image (with those who voted a 1 being dinged more than somehow who voted a 3), but their karma could still go up if they voted the rest of the images fairly.

Another advantage of a karma system is that new users start out with a karma of say 10, meaning that their vote does not weight as heavily as experienced users. This prevents "throw away" accounts from being generated just to give 10's to "friends and family" photos and 1's to every other photo, because the weight of their vote would be lower.

Message edited by author 2004-07-09 19:53:17.

07/09/2004 07:49:54 PM · #14

I can certainly see the merit of the karma system. BTW, thanks EddyG, your explanation cleared up a couple questions for me. That said, it would also certainly complicate the voting statistics. Currently we see a voting histogram, which is massively useful if you really dig into what it is telling you. With the Karma system, every vote now has not only a value (1-10), but also a weight (0-100). You can no longer do a simple histogram, and doing the math at challenge's end gets a whole lot more complicated. No worry about the math, we've got 'pooters for that, but I do think losing the histogram would be a huge loss.

digistoune

07/09/2004 07:53:55 PM · #15

Originally posted by alionic:

Now please don't misunderstand me - I am not whining about my score for the Extraordinary challenge - I only want to use it as an example... That said, I received fifteen 1's, twelve 2's and thirteen 3's for my entry. Do ya'll think it was bad enough to warrant those scores? Personally, when I give a 1 to 3 vote it's because I felt that the photo had enough technical flaws to warrant it or I felt it didn't meet the challenge at all. And I am guilty of not leaving comments to go along with those scores sometimes (I will work on that in the future). But I'm getting the feeling that I should ignore that end of the spectrum regardless of my opinion.

wwwavenger

07/09/2004 08:03:40 PM · #16

::pokes site admins::

07/09/2004 08:12:26 PM · #17

Originally posted by digistoune:

...I received fifteen 1's, twelve 2's and thirteen 3's for my entry. Do ya'll think it was bad enough to warrant those scores? ...

Well, no, as a matter of fact I personally though it was very good, though I didn't get to vote on that challenge. I did run your vote histogram through my statistical analysis, and yes, your image is one which received a few excess ones, and possibly a couple excess twos. Given your score distribution, however, I would have expected you to get at least 8 ones, and a few more 2's. You have a very flat distribution (that is, a large standard deviation); you fell into the "controversial photo" pit. The site is international and not everyone has a positive view of American patriotic images, especially those having directly to do with the military. That is probably why your image generated a more diverse set of votes than some others.

Message edited by author 2004-07-09 20:14:18.

07/09/2004 08:13:12 PM · #18

Originally posted by kirbic:

No worry about the math, we've got 'pooters for that, but I do think losing the histogram would be a huge loss.

I don't see why we would have to lose the voting histogram if we implemented karma-weighted voting.

Instead of the current histogram, I envision say, a 300-pixel wide graph representing 100 possible values (each bar would be 2 pixels wide, with a 1 pixel space). The graph would be labelled 1-10, but there would be 0.1 divisions between each integer (and the left side would start at 0.1 instead of 1, but still end at 10). So someone who voted a "10" on an image that had a karma rating of "50" would show up as a "count" in the "5" bar. Someone who voted a "7" with a karma rating of "80" would show up as a count in the "5.6" bar. So all of the data used to arrive at a score would still be "visible".

Another benefit of a karma system is the concept of "jurors". These would be people are temporarily assigned more than the normal karma maximum (their karma could be the equivalent of 150, for example.) They could be randomly chosen from a pool of voters who have a 100-karma rating and have been consistently active voters in the challenges over the past 3 months. They could be assigned "jury duty" for a 2-week term. These jurors would balance out anybody who is "cheating" by having friends and family vote "high" for their image, for example.

Message edited by author 2004-07-09 20:15:29.

Kylie

Panasonic Lumix DMC-FZ35

07/09/2004 08:16:48 PM · #19

Originally posted by digistoune:

Do ya'll think it was bad enough to warrant those scores?

No, there is no way your work deserved less than a 5, but I think that you were partially hit by the patriotism displayed that some people react against, just as they do religion, etc., in some instances. There was a photo a month or two ago that the general concesus was that the flag hurt it (right or wrong).

EDIT: Kirbic beat me to it while I was interrupted typing. But I see he felt the same thing may have hurt you in this case. What a great technique you used, though!!

Message edited by author 2004-07-09 20:18:36.

07/09/2004 08:19:45 PM · #20

I actually don't think there is any way to implement a histogram that means anything when using the karma system. Your proposal would not actually work, the math doesn't work out.
Bottom line is with karma, youcan't actually assign a single value to a vote, it is in a 2-dimensional space, with dimesnions of score & karma. Both "coordinates" are necessary to calculate the final average.

digistoune

07/09/2004 08:51:41 PM · #21

Aww thanks Kirbic & Kylie - I appreciate your comments. Like I said, I wasn't trying to whine about my score. I like the photo and hoped it would do better, of course, but if someone didn't like it enough to give it a 1 then so be it.

But (if I may quote myself) what about... when I give a 1 to 3 vote it's because I felt that the photo had enough technical flaws to warrant it or I felt it didn't meet the challenge at all. And I am guilty of not leaving comments to go along with those scores sometimes (I will work on that in the future). But I'm getting the feeling that I should ignore that end of the spectrum regardless of my opinion.

Kylie

Panasonic Lumix DMC-FZ35

07/09/2004 08:54:44 PM · #22

Originally posted by digistoune:

I get stumped on the 1-3's myself. It is hard to say, so I won't even try to give advice!