Statistics: It's harder to ribbon...

05/28/2006 12:36:49 AM · #1

I've felt like the cutoff to ribbon has been getting harder. It turns out to be true, but maybe not as much as you'd think.

Method: I took the member challenges from current to January this year and May to January last year. I took out all Free Studies and Speed Challenges and tossed the highest and lowest scoring challenge. All this was to try to minimize the effect of outlying challenges. In a final attempt to do this I looked at the score required for third place. This helps to keep the stats from skewing if someone scores an 8.2 but second is a 7.4.

Results
2006
Average 3rd place score: 7.04
Median 3rd place score: 7.03
(this is good, when median and mean (average) are similar, it means that there aren't a lot of outlying scores.

2005
Average 3rd place score: 6.97
Median 3rd place score: 6.92
(a bit worse, there must be some outlying higher scores which pulled the average up.)

It may not seem like much, but I think a difference in the median of a tenth of a point is significant; especially after I tossed the Free Studies and Speed challenges which seem to score differently than your average challenge.

We seem to believe that in the beginning of the site, the scores were higher. This was true. Then the scoring got harder and the average when down. This is also true. But I believe that having more people around has now made the high scores increase. The average score has also increased:

2006
Mean score: 5.54
Median average score: 5.53

2005
Mean score: 5.41
Median average score: 5.40

Interesting stuff...

Katmystiry

Canon EOS-5D Mark II

Canon EF 85mm f/1.8 USM

05/28/2006 12:40:09 AM · #2

Very interesting .. thank you for all the work figuring this out and sharing!!

deapee

Kodak V803

Sony 24-70mm F4 Vario-Tessar T* FE OSS

05/28/2006 12:40:37 AM · #3

nice work doc

Bear_Music

Samsung Galaxy S24 Ultra

05/28/2006 12:50:55 AM · #4

2005, however, is after (or at the same time as) people started making these claims, as it were. Be interesting to see the 2004, 2003 stats also. Regardless, thanx for crunching, interesting.

Robt.

Mick

Canon EOS-5D

Canon EF 70-200mm f/2.8L IS USM

05/28/2006 12:51:16 AM · #5

LOL! I don't care what the stats say, it's still too easy for a no-talent looser to win a ribbon here. I proved it... twice.

fotomann_forever

05/28/2006 12:51:22 AM · #6

That is interesting.

coolhar

Canon PowerShot Pro 90 IS

05/28/2006 12:58:56 AM · #7

I don't think it's any harder to ribbon now than it has been in the past. People are just voting higher than they used to.

CEJ

Canon EOS-40D

Canon EF-S 18-200mm f/3.5-5.6 IS

05/28/2006 01:02:53 AM · #8

Interesting but not captivating. I can appreciate the work put into collecting the data, but it cannot be accurate as it nowhere takes into account individual improvement/decline curves.

Message edited by author 2006-05-28 01:04:42.

05/28/2006 02:00:51 AM · #9

Originally posted by coolhar:

I don't think it's any harder to ribbon now than it has been in the past. People are just voting higher than they used to.

It's an interesting chicken or egg thing. I've been trying to think of a way at getting at this question, but haven't come up with one yet.

David.C

05/28/2006 02:14:21 AM · #10

Originally posted by DrAchoo:

I've felt like the cutoff to ribbon has been getting harder. It turns out to be true, but maybe not as much as you'd think. ...

Results
2006
Average 3rd place score: 7.04
Median 3rd place score: 7.03
(this is good, when median and mean (average) are similar, it means that there aren't a lot of outlying scores.

2005
Average 3rd place score: 6.97
Median 3rd place score: 6.92
(a bit worse, there must be some outlying higher scores which pulled the average up.)

... I believe that having more people around has now made the high scores increase. ...

I wouldn't put this off as just the effect of more people. A larger population tends to pull the stats in toward the average, not away from it. An interesting analysis would be to adjust the averages for the size of the population -- but it has been way too long since I studied statistics for me to do that. :(

Originally posted by Alienyst:

... it cannot be accurate as it nowhere takes into account individual improvement/decline curves.

Stats are not intended to take into account individuals -- just trends within groups of individuals. The statistics reported show a maturity in the knowledge, ability and skill in this community. But again, that showing of maturity is dampened by the tendency toward the middle as the population increases.

Interesting stuff -- the main thing I'm taking away from it is it shows the maturity of the knowledge, ability and skill in this community is increasing faster than the size of the community is. But I too would like to see the results from the first two years as well -- if for no other reason than to see when it was the maturity first started overcoming the inward pull of a larger community.

David

05/28/2006 02:32:34 AM · #11

Originally posted by David.C:

I wouldn't put this off as just the effect of more people. A larger population tends to pull the stats in toward the average, not away from it.

I agree and disagree. There is a regression to the mean for the average score, but there should also be more outliers which are all ribbons are interested in.

Look at it this way. If you take 10 people, the average IQ may be 110 with nobody above 130. If you take a million people, the average IQ will be 100, but there are bound to be a few with IQs of 150 or more.

A final easy way to think of it is the more people there are the harder it is to come in first...

_eug

Nikon D50

Nikon AF-S DX Zoom-Nikkor 18-55mm f/3.5-5.6 ED

05/28/2006 02:35:56 AM · #12

Alery doctor AND Statistician? What will the think of next?

BeeCee

Nikon D40

05/28/2006 02:41:50 AM · #13

Originally posted by _eug:

Alery doctor AND Statistician? What will the think of next?

Allergy doctor/statistician/photographer maybe? ;)

Message edited by author 2006-05-28 02:42:03.

_eug

Nikon D50

05/28/2006 02:43:32 AM · #14

Originally posted by BeeCee:

Originally posted by _eug:
Alery doctor AND Statistician? What will the think of next?

Allergy doctor/statistician/photographer maybe? ;)

DOH! She got me on that one! lol

David.C

Carl Zeiss 50mm ZE f/1.4 Planar T for Canon

05/28/2006 04:56:44 AM · #15

Originally posted by DrAchoo:

Originally posted by David.C:

I wouldn't put this off as just the effect of more people. A larger population tends to pull the stats in toward the average, not away from it.

I agree and disagree. There is a regression to the mean for the average score, but there should also be more outliers which are all ribbons are interested in.

Look at it this way. If you take 10 people, the average IQ may be 110 with nobody above 130. If you take a million people, the average IQ will be 100, but there are bound to be a few with IQs of 150 or more. ...

Agreed, but it is not a good comparison. A 100 IQ is 'by definition' the average of the population, so the mean IQ will always be 100 regardless of the intelligence of the population. That is, the average of the scale used to measure an IQ is adjusted about the average result of the testing. That is not the case with voting here. The average of the voting scale stays at 5.5, but the average for each challenge varies considerably.

There are more outliers here than there were, but I see that more as a part of the photographic maturity of the site. Between being able to attract photographic talent already developed (Robert and Daniel spring to mind) to the many who have built their ability with sheer determination, hard work and participation in the community. There are more outliers, but their participation is a product of the sites maturity and their participation is helping to mature the site even more.

As I said, it's been years since I studies statistics, but to further support my evalutation of your results ...

Originally posted by DrAchoo:

... A final easy way to think of it is the more people there are the harder it is to come in first...

Yes, harder to reach the top -- but, a larger population does not make it harder to be average. Your results also show an increase in the mean scores between the two test periods. This I take as further support the maturity of the site has increased -- and take it as significant the mean has moved above the scale average.

Regardless, I agree with your conclusion -- it is harder to ribbon.

David

yanko

Canon EOS-5D Mark II

05/28/2006 05:26:34 AM · #16

Someone with a bunch of ribbons and honorable mentions in only 59 challenges telling us it's harder to ribbon? Geesh. I bet if we removed say the top 5 winningest photographers who are currently active from this equation we'd find out it's damn near impossible to win a ribbon period. :P

Message edited by author 2006-05-28 05:27:27.

obsidian

Nikon D80

Tamron AF 18-200mm f/3.5-6.3 XR Di II for Nikon

05/28/2006 05:44:31 AM · #17

Great work Doc!

Just a thought here...

The statistics reveal scoring trends but how would one measure exactly what was influencing a person's scoring? By this I mean what are the things that cause someone to give a higher score to one picture than another?

Some things "appear" to be obvious, such as the (infamous) DPC colours, happy being better than sad (inconclusive), cheese scoring low (sorry!). But what are the other perceptional clues and pointers that cause voters to vote in the way that they do?

Obviously, there are some factors that are probably not discernible and therefore measurable - mood, mental disposition at time of voting, calibration of monitor - but I do wonder what the others are and whether there are any detectable statistical trends.

Carl (Sunday morning, sinus headache, monitor in low light condition, coffee consumed)

zxaar

05/28/2006 08:20:30 AM · #18

i do not understand why it is harder to ribbon, same as old times every challenge has ribbon winner. Its as easy as old times. Only scores have dropped. So what, we still have winners, don't we.

joebok

Fujifilm X100F

Nikon AF-S DX Zoom Nikkor 18-135mm f/3.5-5.6G IF-ED

05/28/2006 08:26:37 AM · #19

Originally posted by DrAchoo:

Originally posted by coolhar:
I don't think it's any harder to ribbon now than it has been in the past. People are just voting higher than they used to.

It's an interesting chicken or egg thing. I've been trying to think of a way at getting at this question, but haven't come up with one yet.

I've always thought a re-vote on old challenges would be an interesting measure. There would be some noise - the top images would probably be recognizeable for a chunk of people, I would think that would tend to make the scores on them a bit higher (as people like to vote for winners) - but I would still expect some useful "vote inflation/deflation" data to be extractable.

coolhar

Canon PowerShot Pro 90 IS

05/28/2006 09:00:17 AM · #20

I think some of you are ignoring the most obvious answer. I think that people tend to vote higher these days because of all the negative forum threads and posts about low votes, and voters. Think about how much talk there has been using terms like "trolls" and "nazis" to describe people who cast low votes. And hardly ever a word complaining about anyone voting too high. The suggestion in the rules to comment when scoring a shot 1, 2, or 3; complaints that the suggestion is not complied with; the posts about how there are lots of great photos in a challenge; the permissive attitude toward sharing entries for "helping" others to learn; the unofficial teams competitions; calling people out for having a low Avg Vote Cast stat in their profile; etc. It all mounts up, and, over time, pushes scores upwards. There is little or no counterbalancing influence pushing scores downwards.

Thanks for compiling the numbers for us Doc.