What on earth do I make of this score distribution

03/19/2009 02:08:58 PM · #51

Originally posted by mikeee:

Originally posted by kirbic:
In nearly all cases, voting closely approximates a Gaussian distribution, usually with a very few extreme outliers.
In this specific case, there are four outliers (all of the votes of "10") and if those are removed, the distribution matches a Gaussian curve with a r-squared value of 0.998.

So we could have 2 results, the current one and the one which excludes outliers individually until it achieves a target r-squared value, such as 0.995 or 0.997?

Yeh, that's the cynical approach :-P
I'd go into how one determines analytically which data points can be regarded as outliers, but I doubt anyone would listen. Trolls explain everything much more easily.

posthumous

Canon EOS-40D

Canon EF 50mm f/1.8 II

03/19/2009 08:14:12 PM · #52

Originally posted by kirbic:

Originally posted by posthumous:
the problem with all complaints about "distribution" of votes is that the notion of a Gaussian or any sort of balanced distribution does not apply to voting. Why should it? Read Taleb's The Black Swan to learn more about this.

More appropriately, "why shouldn't it?"

the burden of proof is on you, not me, but i do have reasons why it shouldn't. votes are not random, nor are they based on some biological imperative like height.

and the fact that you can find similarities in the data because that's what you're looking for, and because you assume something about regions less than 1 and greater than 10, regions which do not exist and whose existence would change the very nature of the scale, is not very impressive. It's reminiscent of people who think swans are white.

03/19/2009 08:35:36 PM · #53

Why oh why does everyone think everything should be Gaussian? Where was this taught? That is so far from the truth. First of all, Gaussian is one (not the only) type of distribution related to random events. There are other distributions. There are types of events/data that aren't random.

Voting by nature is biased. This bias may lead to extreme distributions or more than one spike.

03/19/2009 10:05:42 PM · #54

Let me restate my case and hopefully correct some misconceptions.

1.) I did *not* say that all random process produce Gaussian distributions. Far from the truth. Gaussian distributions do occur with amazing regularity, but there are many other possibilities.
2.) Voting for each individual user is biased. When a large number of (diffrerently) biased voters use the same scale (differently for each user) the result is, in fact, a close approximation of a Gaussian distribution.
3.) Biased voting, where a significant group of the voting populations is biased in a particular manner, *can*lead to odd-shaped distributions. This does happen here at DPC, very occasionally, on images that produce strong reactions in some groups but not others For most images, this is not the case.
4.) If "image A" has a distribution of votes that is Gaussian, and if history has shown (it has) that the distributions for most DPC images are close to Gaussian, I think it is fair to conclude that image A's distribution being Gaussian is not due to chance, but to the underlying nature of the voting.
5.) I will *not* prove that the distribution for this or any other image is Gaussian. That is trivial mathematics, and anyone not too lazy to do so can do it for themselves. Why would lie about such a thing?

Originally posted by posthumous:

the burden of proof is on you, not me, but i do have reasons why it shouldn't. votes are not random, nor are they based on some biological imperative like height.

You really don't know a thing about statistics, and yet you'd lecture me. Anything that doesn't fit your personal world view is rejected with disdain, because it hasn't been "proven" to you. No, see 1-5 above. Go and prove it or disprove it yourself, or take the data at face value.

03/19/2009 10:17:31 PM · #55

Originally posted by mshimer5:

I am gonna weigh in to support Fritz (Kirbic) here:

Nobody, as far as I know, and certainly not Fritz, is saying the distributions *have* to be gaussian. What he is saying is that he has studied the distributions *extensively* (probably more extensively than anyone else in DPC, I'd wager), and that what he has observed is that in the vast majority of cases, the distributions *are* gaussian.

Anyone who can't make this distinction really shouldn't be discussing statistics at all, frankly.

R.

Message edited by author 2009-03-19 23:20:31.

03/20/2009 12:43:46 AM · #56

Originally posted by Bear_Music:

Originally posted by mshimer5:
Why oh why does everyone think everything should be Gaussian? Where was this taught? That is so far from the truth. First of all, Gaussian is one (not the only) type of distribution related to random events. There are other distributions. There are types of events/data that aren't random.

I am gonna weigh in to support Fritz (Kirbic) here:

Nobody, as far as I know, and certainly not Fritz, is saying the distributions *have* to be gaussian. What he is saying is that he has studied the distributions *extensively* (probably more extensively than anyone else in DPC, I'd wager), and that what he has observed is that in the vast majority of cases, the distributions *are* gaussian.

Anyone who can't make this distinction really shouldn't be discussing statistics at all, frankly.

R.

If that is not what was being said, then sorry...it is just so often people automatically assume Gaussian, and it drives me crazy. But, to make a point: as you said and he pointed out, maybe not for the majority of the time, but I would guess quite often the distribution is not gaussian. This is due to different peoples scales in voting. Not everyone is technically using the same scale, thus you should *not* assume gaussian even if it happens with a ton of regularity. I just don't think central limit really applies here. I admit I may be nit picking here, but statistics is my job, so when someone incorrectly analyzes statistics, it jumps out to me.

What I find more interesting (and this is quite possibly the original intention) is how people's differing scales leads to these non-gaussian distributions. It obviously has something to do with the subject, but what exactly? And what would happen if you renormalized everybody's votes that they give per challenge?

03/20/2009 12:55:21 AM · #57

Originally posted by mshimer5:

Fair enough. Just be aware that Fritz knows his statistics as well; his work demands it. And he, when he was on site council (for many years) was our resident statistics guru, always around to keep people from making outrageous assumptions using "lies, damned lies, and statistics." As long as you're aware that he's not making statements like "HAS to be gaussian" but instead is making observations based on many hundreds of analyses of voting distributions, that the gaussian distribution is by FAR the most common in DPC voting, then everything's jake :-) Sorry if I jumped all over you...

R.

Sony SAL-1855 DT 18-55mm f/3.5-5.6

03/20/2009 12:58:43 AM · #58

Originally posted by Bear_Music:

No problems...I took no offense. Just a miscommunication on my end - the bane of forums. I'm just enjoying discussing statistics that aren't related to my research for once.

lawrysimm

Sony SLT-A57

03/20/2009 04:20:43 AM · #59

Quite aside from the debate over the golden retriever shots... there is some excellent advice on colour balance in this thread which I found really useful. And no disrepect to the photographer, but I prefer the balanced versions by Bear_Music and jeger. Both photo's look more natural which I think does indeed indicate an original problem with the WB.

I gave the image a 6 in voting, and would probably have given a 7 to it if the colours looked more natural as per the reworking in this thread.

lawrysimm

Sony SLT-A57

Sony SAL-1855 DT 18-55mm f/3.5-5.6

03/20/2009 04:22:01 AM · #60

Originally posted by Bear_Music:

Yeah but don't forget that 94.8% of all statistics are made up on the spot.

03/20/2009 07:54:00 AM · #61

Originally posted by LadyStarstruck:

Originally posted by Bear_Music:
Fair enough. Just be aware that Fritz knows his statistics as well; his work demands it. And he, when he was on site council (for many years) was our resident statistics guru, always around to keep people from making outrageous assumptions using "lies, damned lies, and statistics." ....
R.

Yeah but don't forget that 94.8% of all statistics are made up on the spot.

Damn, she got me... ;-)

03/20/2009 08:01:11 AM · #62

Originally posted by LadyStarstruck:

I thought that was 98.6%? No, wait, that's human body temperature... Or is it? According to the good folks at Harvard, it should be 98.2%...

R.

03/20/2009 08:06:00 AM · #63

Originally posted by mshimer5:

What I find more interesting (and this is quite possibly the original intention) is how people's differing scales leads to these non-gaussian distributions. It obviously has something to do with the subject, but what exactly? And what would happen if you renormalized everybody's votes that they give per challenge?

Ah, but don't assume that peoples' voting bias *will* lead to non-Gaussian distributions. It often won't. Sometimes it will, and the *why* can be pretty informative. For most images, I believe that central limit *does* apply because for the majority of voters, the lifetime vote distributions are Gaussian. I have proven this to myself by actually running the tests on peoples' lifetime distributions, including a few of us who have voted on 100,000+ images.
For some images, I've seen vote distributions that are strongly non-Gaussian, and even seen a little bimodality creep in. These shots tend to be those that strike a strong reaction in a particular group. That group then votes very low (or high, but it's usually low) and the distribution becomes distorted. In general, the greater the spread of the (approximately Gaussian) distribution for a photo, the more disagreement about the quality of the shot. It's quite interesting to look at voting distributions and try to draw conclusions as to *why* they are what they are.
Even in the case where a distribution is "truncated" by being pushed up against the limit of the voting scale, with a spike of votes at either 10 or 1, the fit to a Gaussian distribution can still be assessed, and is quite often still very good.

03/20/2009 08:14:31 AM · #64

It occurs to me that a lot of people may have no idea what we're talking about with "gaussian distributions"... For what it's worth, the standard "bell curve" is an example of gaussian distribution. Wki has a short, simple overview here.

R.

03/20/2009 10:38:56 AM · #65

I don't doubt that many (practically all) will be nearly Gaussian. My point was that one shouldn't be surprised if they find an image that isn't Gaussian. As you say, people's individual voting habits will be Gaussian (that is easy to see), but what is more difficult is that the distribution of all individual Gaussian must itself be Gaussian in order to expect Gaussian all the time. Certainly *not* trivial. So, even though most are Gaussian, I am not convinced (mathematically speaking) that the central limit applies, as everyone is pulling from there own, unique, random distribution.

Out of curiosity, have you attempted to fit any of the data to other distributions, say Poisson?

posthumous

Canon EOS-40D

Canon EF 50mm f/1.8 II

03/20/2009 01:19:44 PM · #66

Originally posted by kirbic:

I don't know a thing? So a random sampling of people's heights will not produce a Gaussian curve? I read a book called the Black Swan, so I must know something. I recommend it for all statisticians. One of the things it mentioned was very smart people with lots of knowledge and expertise saying things like "blue chips are a safe investment" because the data supported that idea. This is where your number 4 comes in, and that's what concerns me, the idea that voting has a Gaussian "nature." That sounds pretty vague to me, but the way I best interpret it is that there is one score that is most normal (sorry if "normal" has technical connotations that don't apply here) and then other scores diverge from that normalcy, quickly dropping off. Think of the rareness of someone who is 10 feet tall. The implication is that each picture has an intrinsic worth which a sheer mass of collected and analyzed opinions can divine. When voting curves deviate from Gaussian, it can be attributed to some sort of bias. The dichotomy of normaal vs. biased does not appeal to me. It has had many unfortunate applications through history.

It is interesting that you found the distributions to be Gaussian, I trust you on that, and the fact should be considered by anyone who is going to develop a theory of voting on dpchallenge (god bless whoever that is), but I still worry about the truncation. I guarantee you that if you change the scale to 20, you will not see the curve graciously smooth out to Gaussian perfection. Instead, you will see a similar "truncation." There is no law preventing people from being 300 feet tall. But there is a law preventing people from voting 300 on a photo. My personal theory is that a larger scale would lead to results that are less Gaussian-looking, but we've no data.

p.s. one must also consider the importance of the divergences that do occur. For example, there are many cases of small spikes on the extreme low and high scores. This doesn't look very significant, but it is very anti-Gaussian, where outliers are supposed to become exponentially rarer as they go further afield.

03/20/2009 01:36:16 PM · #67

how would a voting scale of 1-20 affect the curve of the results ?

how would a scale of 1-100 affect the curve of the results?

you'd get a wider distrubution of votes over the whole of the scale, and a less obvious peak in the curve. but outside of that ?

i'm not an expert - and haven't read you're suggested book. but i don't see your logic.

Sony FE 200-600mm f/5.6-6.3 G OSS

03/20/2009 01:49:01 PM · #68

I am unsure if this is what he meant, but theoretically changing the scale from 1-10 to 1-100 would be making the variable a little more continuous. This would increase the resolution of your data and you may see extra spikes in your data. However, I do not see how it would affect the truncation. All it should do would be increase the resolution (and even that wouldn't be much).

PapaBob

Sony Alpha a7 IV

03/20/2009 01:52:10 PM · #69

Not to mention the 1-100 scale would make the new 1 the equivalent of getting a .1 on the current scale, if I get a .1 then I am really gonna be pissed off.....:P

03/20/2009 01:56:19 PM · #70

Originally posted by posthumous:

This is where your number 4 comes in, and that's what concerns me, the idea that voting has a Gaussian "nature." That sounds pretty vague to me, but the way I best interpret it is that there is one score that is most normal (sorry if "normal" has technical connotations that don't apply here) and then other scores diverge from that normalcy, quickly dropping off. Think of the rareness of someone who is 10 feet tall. The implication is that each picture has an intrinsic worth which a sheer mass of collected and analyzed opinions can divine. When voting curves deviate from Gaussian, it can be attributed to some sort of bias. The dichotomy of normaal vs. biased does not appeal to me. It has had many unfortunate applications through history.

I don't see that implication at all. I think you're reading too much into what Fritz is saying. I don't think the gaussian curves, the bell curves, that we see so often in DPC are any measure of an image's "intrinsic worth" at ALL, myself. No, what they are measuring is that in a broad enough sample there will be a distinct pattern to how the scores lie around the peak values, and we will see an aggregate of how the voters responded to an image.

And that's IT...

When the peak is shifted way to the right, and the other votes congregate mostly to the right, then we have and indication that the voters, in the aggregate, responded very, very favorably to the image. Shift the peak way to the left, the opposite is true. Has nothing to do with intrinsic worth, whatever that is. Show the same images to a bunch of nursery-school kids and the distribution would be different, and THAT wouldn't define "intrinsic worth" either.

In a scientific sense, the term "bias" is not a negative one. It just indicates a condition where one edge of the scale, or aspect of the thing being studied, is favored over the other/s.

Take a look at the curve on this one, when the site was much smaller than it is now:

R.

Message edited by author 2009-03-20 14:00:24.

03/20/2009 01:57:30 PM · #71

that's my point. and how i took what he said.

as far as some newfangled way of actually voting - and possibly acheiving posthumous' goal.

have three levels of a 1-10 scale. everyone would cast three votes per image - instead of one. probably not going to happen.

level 1 - being - meets the challenge theme.
level 2 - being - it's aesthetcally pleasing.
level 3 - being - it's technically sound.

or something along those lines.

that would give three different curves to look at - each potentially peaking in a 'non-normal' way. those scores averaged out to a final score would likely still fall the way the general curve falls now...

03/20/2009 02:00:21 PM · #72

very true - in the sense mentioned by posthumous - bias is a political term :)

Originally posted by Bear_Music:

In a scientific sense, the term "bias" is not a negative one. It just indicates a condition where one edge of the scale, or aspect of the thing being studied, is favored over the other/s.

03/20/2009 02:06:29 PM · #73

That is precisely why I am not too sure about the central limit here. If you look at the derivation of the central limit, it assumes some intrinsic mean with some finite variance. (yes, i know you don't *need* an intrinsic mean for central limit, but without it, the application is far less trivial) I do not believe there is any intrinsic mean. As you said, different groups will offer completely different distributions. I wager that when looked at at different times you also could get different distributions. Now, I admit I haven't collected any data on the subject, so I am only speculating. I would love to see how these trends evolve: i.e. how does the variance change as the number of votes increase? Is it constant or does it change (up or down)?

03/20/2009 02:18:31 PM · #74

as it stands now - i think the voting scheme is designed to give a reasonably accurate account of the overall view of the voters - while at the same time keeping it simple enough to attempt to entice people to vote. if the voting methods were too convoluted - you'd likely see less votes and actually see less accurate results - as a result.