Hermit
Archon
Posts: 4287
Reputation: 8.94
Rate Hermit
Prime example of a practically perfect person
 
|
 |
Suggested alterations for comment prior to list posting
« on: 2003-08-15 08:57:39 » |
 |
Having observed the extreme list volatility of late, last night, as an experiment, to attempt to establish what others thought of one another and in an attempt (so far as I can) to order the list in how I would value people, I varied the reputation values I had assigned from what I really think people deserve in terms of the "Rating Key", i.e.
1# This person should be banned
2# Little or no redeeming value
3# I tend to disagree with them
4# Slight negative opinion
5# Neutral or no opinion
6# Slight positive opinion
7# I tend to agree with them
8# Generally excellent contributions
9# A pillar of the community
to the values needed to achieve the list ordering I desired. I was, to put it mildly, quite surprised by the huge offsets I needed to apply to modify some of the results, not because it was mathematically non-intuitive (it isn't), but because it indicated that people were not using the above rating system as expected.
I suspect that more than one person is doing as "normal" what I did as an experiment, and further suspect that the technique I used may be one being employed by others. If that is the case, then I suggest that those doing so rethink their strategy, as I suspect that this practice is harmful in a number of ways. Here is why (bear in mind that the numbers relate to a temporary situation as I did them while I had the list reordered - but they will serve for examples).
1) It creates very high list volatility, needing a lot of attention, because as each person attempts their "restructuring" so the others have to respond to restore it to theirs.
2) It results in extreme values being assigned to people - at both ends of the scale - completely distorting both the balance and the meaningfulness of the system.
3) I suspect that the Reputation Index does not accurately reflect the values above. Do we really "tend to agree with David Lucifer" as his 7.3099 rating implies? Or is he more accurately, a "Pillar of the community?" Do we have a "slight positive opinion" about ElvenSage as his 5.9607 suggests? Or is he worth somewhat more? Do we really have a "Neutral" to "Slight Positive" opinion of Michelle as Reputation Index would suggest?
4) The fact is that most people are rating towards the upper end of the curve (as evinced by the fact that the lowest rated person, Joe Dees, is seen as 4.5361 (or slight negative to Neutral or no opinion rating) and because of that, it is easier for other members to affect somebody else's position on the ratings by dragging them downward (a range of 4 choices including neutral) than bumping them up (a range of 4 choices but closer to the average value). An few examples may suffice to explain this better:
Let's look At David Lucifer, who surely has a mean score of 9. He can't be bumped up, as he is already "at the top of the scale." So other users can only move him down, the question is by how much. Any one user has the ability to change his score only a little. Lets say an average user has an "equity" of 2.5%. Then the greatest change they could effect would be 0.8 points downward, multiplied by the "equity" that their ranking provides. i.e. 8 x .025 = a rating change of 0.2 at most. Lucifer with 10% odd "equity" could downgrade his "reputation index" (directly) by 0.836. He could make his influence somewhat greater by reducing the influence of others - taking his consequently lowered reputation as a side effect, but definitely changing the list order.
Now lets look at Michelle, who at rank 17 is close to the middle of the rankings. Her "reputation index" of 5.6841 is probably made up of many Neutral votes, and a few more extreme votes to "pull her up" or "push her down". I say extreme, because the averaging effect means that any one voter's ability to "move" the position of any similarly positioned member is fairly limited. One of her peers (i.e. with an "equity" of around 2.5%) will likely need to use either a "1" or a "9" to have any significant effect on order and even then, at best, could shove her "ranking" by at most 11th and 22nd positions. Or, approximately, increase her reputation 5 positions or decrease it by 6 positions, but more likely a lot less, given that the further from the mean you attempt to drive her, the more difficult this is.
5) If you rank somebody away from where you think their score should be, I think only a few options exist to explain this:
You don't understand the system
You are being irrational
You understand the system and are trying to impose your preferences on the result, by hypocritically making assertions about others you yourself do not believe.
You are attempting to retaliate (tit-for-tat) to a perceived problem with somebody else's scoring.
I started writing this as an appeal to reason. As I reached the conclusion above, I decided that the system as currently implemented may be an open invitation to a poisoning of the commons and as such, faulty (but not very). As such, I would like to propose a small change to the scoring method.
1) Calculate the modal reputation score for each participant (i.e. the rating integer which received the most selections).
2) Calculate the difference between each person's vote for everyone else and the calculated values.
3) Award a correction factor, perhaps termed a "group incompatability index", to each voter based on the square of the sum of the differences determined in 2. The correction factor should be normalized to a scale equivalent to some percentage of the total range of awarded reputations and then deducted from the voter's reputation index as calculated in 1 before performing the remaining calculations.
4) Distribute and weight the reputations and normalize the results to calculate significance.
5) Calculate equity.
It may be a good idea to show the amount of the correction factor in the "ranking table" in order to encourage users to maintain a realistic value, and allowing others to recognize where a member is accepting a significant penalty to themselves to disagree with the group. Alternatively the difference in ranking might be shown against each ranked person on the "rank voting page".
As a final comment, I suggest a minor change to the wording on the rating key, as while the outer positive values relate to significance to the list, the outer negative values appear to relate to the significance to the CoV (a better scale IMO), and the inner values relate to your opinion of the opinion of the person rated (a valid but I think less useful measure - and definitely not the same measurement as the outers). So I recommend the following scale:
(Less value to the CoV (note that this could be on BBS, IRC or real life)
1# This person should be banned
2# Little or no redeeming value
3# Strongly negative opinion
4# Slight negative opinion
5# Neutral or no opinion
6# Slight positive opinion
7# Strong positive opinion
8# A credit to the the community
9# A pillar of the community
(More value to the CoV - on BBS, IRC or real life)
Hermit
|
Author
Logged
