New judging system suggestion (not easy I suspect)

Sony 24-70mm F4 Vario-Tessar T* FE OSS

02/02/2006 07:40:42 PM · #1

I wrote this in response to a bitch thread on getting 1,2,3 votes but decided to start a new thread on a suggestion.

A number of folks have complained about how a blue ribbon winner can get a score of 1, 2, or 3. This got me thinking about a different method of judging a challenge. What if you had a voting system that would force the voters to set an absolute ranking of every picture in the challenge? Each picture would be compared with a previously "ranked" picture and the voter would select "better" or "worse" thus moving the picture into its proper order.

So the start would be the first picture you see which is #1. Then the next picture you decide if it is better then the first which would then set its position in the queue. Once you find the point where the picture is "better" then the one being compared against, the voting would move on to the next picture. It would require that the voting system display two pictures at a time (and maybe three).

The scores then could be arranged 1 to 10 with the number of entries determining the number of points used in this spread. This would eliminate the "I got 1, 2, 3 votes on my 6.6 average image" complaints because those voters who do that wouldn't really be able to do that since they would have to give the picture an absoulte rank.

Anyway, just a thought... maybe if I get through my startup company I can start a new photo challenge website ;-).

Bear_Music

Samsung Galaxy S24 Ultra

02/02/2006 07:45:05 PM · #2

This is falwed; let me tell you why. Let's assume the first picture you see is, in fact, the second-best in the challenge. And let's assume the second picture you click on is by far and away the best picture in the challenge. End result: EVERY picture you "demote" as not being equal to the "best" picture is ranked ahead of the first picture, which is in fact the second-best picture.

Serious ranking involves a huge amount of back-and-forth, and most of our members don't have that sort of time available.

R.

Prof_Fate

02/02/2006 07:55:57 PM · #3

I like the way fujimugs.com does it-
as a voter you look at the images. you can of course comment as you go.
If you like an image, you add it to 'your favorites'.
one you have looked at all (or some) of the images you go to 'your favorits page' and there sit the images you selected earlier.
So far so easy!

Now you sort them 1-10. the best you rank 1, etc till 10. If you have more than 10 images, number 11 onward just fall off the list.

At the end of the voting period, the results are tallied - a 1 vote is 10 point, a 2 is 9 points, etc. Image with the most points wins.

No more 1, 2, 3 votes. No more 'does not meet challenge' bashing.
If all images get at least one vote, you still get a brown ribbon finisher.

YOu still have winners, rankings, and average points received, % finish, etc. like now. In addition, you get a 'total points earned' count as well.

Just my 2 cents.

conglett

Nikon D70

Nikon AF-S DX Zoom-Nikkor 18-70mm f/3.5-4.5G IF-ED

02/02/2006 08:52:54 PM · #4

Unfortunately, no matter what system for voting is used - it will not be perfect. People will always complain no matter what, therefore I favor keeping the voting as simple as possible.

The way it is right now, it distributes the small amount of "troll" votes fairly evenly.

Canon EF 24-70mm f/2.8L USM

02/02/2006 09:52:15 PM · #5

Originally posted by Bear_Music:

I think I didn't explain this well enough. The comparisons keep moving down the queue until you find a picture that is not better then the current one you are evaluating.

Let's assume there are 5 pictures in the challenge and we will name them pictures A, B, C, D, E.

- The first Picture A gets put into the queue: (A)
- Picture B is better then A so now the queue is (BA)
- Picture C is compared to B (first in queue) and deemed worse. Then it is compared with A and is deemed worse. Since this is the end of the queue the new queue is: (BAC)
- Picture D is worse then B, worse then A, better then C for a queue: (BADC)
- Picture E is better then B for a queue: (EBADC)

Now the voter has a natural ranking of all the pictures and the system can allocate points appropriately.

troylox

Canon EOS-7D

02/02/2006 09:57:38 PM · #6

I don't really see a big problem with the current voting method. I am in my second Statistics class now in the last six months, and through some of what I have learned, those low outlying scores (1,2,3's) on a highly rated photo should not pull down it's rank that much. Every challenge I enter I expect to get a couple low votes. That's why my goal is to submit a good enough photo so that the higher scores will outweigh the inevitable lower scores. That's my take at least. The good photos will still do well, no matter what, even if they do get a few low scores.

Sigma 18-200mm f/3.5-6.3 DC OS for Canon

02/02/2006 11:04:13 PM · #7

That's true... I have to remember my stats classes (from 20 years ago :-( )...

magnus

Canon EOS-30D

02/02/2006 11:53:50 PM · #8

Originally posted by troylox:

Sounds like you are assuming normal (Gaussian) distribution of scores. That's often true, but the score histograms for some images are distinctly bimodal. Those love-it-or-hate-it images (often political/religious images, and some nudes) can get a peek in the high range and another in the very low range. A peek like that down in the 1s, 2s, or 3s obviously has a big effect on the final score. I'm guessing that the fujimugs.com system would boost images like that.

paddles

Pentax K10D

Pentax FA 50mm f/1.4 AF

02/03/2006 04:22:50 AM · #9

Originally posted by dleach:

I think I didn't explain this well enough...

Let's assume there are 5 pictures in the challenge and we will name them pictures A, B, C, D, E.

- The first Picture A gets put into the queue: (A)
- Picture B is better then A so now the queue is (BA)
- Picture C is compared to B (first in queue) and deemed worse. Then it is compared with A and is deemed worse. Since this is the end of the queue the new queue is: (BAC)
- Picture D is worse then B, worse then A, better then C for a queue: (BADC)
- Picture E is better then B for a queue: (EBADC)

In theory, you could do something like this - in practice it becomes very difficult once you have more than a handful of items to rank. Full marks for creative thinking, but when you're talking about ranking 50-400 photos in a challenge, it becomes infeasible. If you want to know why, read on, otherwise skip the rest of the post.

WARNING: LONG POST, CONTAINS MATHS.

The more items you've voted, the longer the next vote takes to do. For the 100th picture you view, you could be making anywhere between 1 and 100 decisions for that picture, just to cast a single vote - on average it would take 50 decisions.
1st picture - 0 decisions
2nd picture - 1 decision
3rd picture - 1 or 2 decisions, average 1.5 = 2.5 decisions so far (on average)
4th picture - 1-3 decisions, average 2 = 4.5 decisions so far (on average)
5th picture - 1-4 decisions, average 2.5 = 7 decisions so far (on average)
6th picture - 1-5 decisions, average 3 = 10 decisions so far (on average)
It gets uglier from there... by the time you get to the 100th picture, you'll have made around 2500 decisions, on average.

You could make it more efficient by using a "divide and conquer" technique (binary search for the comp sci geeks), where you compare the next photo to the one in the middle of your votes so far. Suppose it was better than the middle one, then you'd compare it to the one that's halfway between the middle one and the top (i.e. at the 75% mark). If it was not as good as that, then you'd compare it to the one halfway between the middle and the 75% mark (i.e. the 62% mark) and so on until it slots into place. This would give you
1st picture - no decisions
2nd picture - 1 decision
3rd picture - 2 decisions
4th picture - 2 decisions
5th-8th pictures - 3 decisions each
9th-16th pictures - 4 decisions each
and so on. This isn't quite so ugly, but to vote on 100 pictures requires almost 500 decisions. (Someone who has done maths and algorithmic complexity more recently than me might want to check my logic and maths)

The next refinement would be displaying the ranked queue of ALL the photos you've seen so far, and deciding where the next one fits in. This would reduce the number of decisions to make, but it'd get to be a horribly big page to scroll through and require some, err, creative coding of the web page.

And of course, at the end, the results would be evenly distributed across the scoring range. So inevitably for each person who voted, 10% of their votes cast would be 10, 10% would be 9, etc. etc. 10% would be 1. And every time someone placed a vote, ALL of their votes they had cast so far would have to be recalculated because some of them would be moving up or down a place. It would also change the pattern of final scores - I expect that the top scores in each challenge would be higher than present, and the bottom scores would be lower - you could regularly have people scoring in the 8.5-9.5 range, and in the 1.5-2.5 range, which I think is pretty rare at the moment.

Message edited by author 2006-02-03 04:29:30.