BoingBoing points to a nice pair of articles by Annalee Newitz on how people manipulate online reputation systems like eBay’s user ratings, Digg, and so on.
There’s a myth floating around that such systems distill an uncannily accurate folk judgment from the votes submitted by millions of ordinary citizens. The wisdom of crowds, and all that. In fact, reputation systems are fraught with problems, and the most important systems survive because companies expend great effort to supplement the algorithms by investigating abuse and trying to compensate for it. eBay, for example, reportedly works very hard to fight abuse of its reputation system.
Why do people put more faith in reputation systems than the systems really deserve? One reason is the compelling but not entirely accurate analogy to the power of personal reputations in small town gossip networks. If a small-town merchant is accused of cheating a customer, everyone in town will find out quickly and – here’s where the analogy goes off the rails – individual townspeople will make nuanced judgments based on the details of the story, the character of the participants, and their own personal experiences. The reason this works is that the merchant, the customer, and the person evaluating the story are embedded in a complex, densely interconnected network.
When the network of participants gets much bigger and the interconnections much sparser, there is no guarantee that the same system will still work. Even if it does work, a large-scale system might succeed for different reasons than the small-town system. What we need is some kind of theory: some kind of explanation for why a reputation system can succeed. Our theory, whatever it is, will have to account for the desires and incentives of participants, the effect of relevant social norms, and so on.
The incentive problem is especially challenging for recommendation services like Digg. Digg assumes that users will cast votes for the sites they like. If I vote for sites that I really do like, this will mostly benefit strangers (by helping them find something cool to read). But if I sell my votes or cast them for sites run by my friends and me, I will benefit more directly. In short, my incentive is to cheat. These sorts of problems seem likely to get worse as a service grows, because the stakes will grow and the sense of community may weaken.
It seems to me that reputation systems are a fruitful area for technical, economic and social research. I know there is research going on already – and readers will probably chastise me in the comments for not citing it all – but we’re still far from understanding online reputation.
The problems with spam and with reputation systems come from the low cost operation and lack of damping. Stock markets are even susceptible (pump-and-dump scams hype penny stocks in exactly the manner described earlier) and have damping in the form of brokers charging for every transaction. But the hyping has very low transaction costs, particularly if done by e-mail spam, as is usually now the case.
The solution is to make some online activities, such as sending an unsolicited mail (recipient doesn’t know you) or voting for something, have a higher transaction cost. Design into the protocols a mandatory piece of computational make-work per vote or per recipient, such as to crack a 64-bit cypher (easy enough but takes some CPU time). (The bit size must be raised gradually over time). An easy way is for the new protocols to use encryption at a deliberately shoddy bit length and not provide decryption keys, so to join the session requires brute-forcing the encryption, but it’s doable. An ordinary user won’t notice a few seconds’ delay per recipient per email or per click on a “+” or “-” button on some site’s voting page, but it sure will slow down spammers and ballot box stuffers! Other options include “funny money” systems — give users of a system credits to spend. Sending/posting messages and casting votes uses credits. Credits regenerate automatically, but slowly. Members with solid reputations regenerate credits more rapidly. But also get their regeneration rate knocked down further if they are abusive and get many votes cast against them. Getting new accounts means being stuck with the slow initial regeneration rate, and may require a captcha or brute-forcing a small cypher or both, raising the transaction cost of whitewashing your identity. Bad enough reps of course lead to a regeneration rate of zero, or nearly so, locking you out at least for a while, or forcing you to jump through whatever computational hoops to get a new account. This gives second chances but greatly limits abuse. Of course, it requires new protocols and client software for a variety of things, followed by a transition period of phasing out older protocols (such as SMTP)…
March 2007 Quick Links Part 2…
By Eric Goldman Yesterday I posted the Google edition of my list of interesting items from March. Today I post……
Fine article, fine comments.
The one thing I’d add is that the anonymous nature of many reputation systems leaves the items they are recommending open to market for lemons problems. This is one more reason to doubt the ability of recommendation systems to handle niche products as well as they handle “hits”.
“This means that price movements are often exagerated, and it is often possible to make money on the “dead cat bounce†which is simply an artifact of overdamping in the system.”
Overdamped systems do not bounce. You probably were trying to say “underdamped”.
The Wired example of Digg manipulation showed that in the early stages of voting (once a site reaches critical mass) a positive feedback takes over as more people vote for no other reason than the apparant “rising star” status (not for any intrinsic property of the site). This is exactly the same positive feedback that happens with penny stocks that get a sudden burst of publicity. In both the Digg example and the stock market — what mechanism provides a damping effect?
Looking at the theory of control systems, damping is a force that works in opposition to change in the system. Friction is an example in mechanical systems — whichever way you want to push the box, friction will always push against you. In electrical systems resistance provides the damping.
In financial systems, transaction taxes are the equivalent of damping — they discourage anyone from changing the system state. In reputation tracking systems, casting a vote is a very low-cost operation (by design) and there is no penalty for getting it wrong, thus the system has no damping. Hardly surprising when positive feedback takes over and leads to instability.
My twenty-something friends and I rely heavily on Internet reviews of products, especially electronics. Because we’re aware of the hyping up that goes on, we talk to each other a lot to find out if anyone we know has the product in question. Or has tried out the product. And rely heavily on review sites like E-Opinions.com. I like to think I can tell when a review is fake, and I concentrate on specific descriptions. (Like, the product makes too much noise, or stopped performing a specific function within six months).
The problem is, when there are no reviews of one product and say, two reviews of another, similarly-priced product, I consistently find myself buying the product that has SOME reviews rather than none. (Knowing all along that the existence of two reviews doesn’t make the product superior).
That didn’t add much to the conversation, did it. @_@
I have a theory that this is why people love Oprah. With so many brands to choose from and no idea what the differences are, it’s nice to have someone you trust make the decisions for you. Read these books, use this lotion, eat these chocolates…if Oprah says it’s good, it’s all good. Makes life easier.
It’s got to be linked to fear of being taken for a fool, for paying too much for a product or finding a “deal” only to discover the product is worthless. (The difference being paying $100 for a product you could have bought for $25 elsewhere, or paying $25 for a product that’s really worth $25..but a VCR that’s worth $25 is not good for much, and you should’ve just paid $100 for a better one.)
For some good examples of the problems with reputation systems consider the New York Stock Exchange, or any stock exchange for that matter. The price of stocks depends on the reputations of the companies involved and their peers, and the reputations of the analysts reporting their reputations. The purpose of the exchange is to establish a current concensus on the reputations of the companies being traded.
One problem is that the reputation makers have a vested interest in providing honest assessments. Naive investors tend to believe that stock exchanges are used to raise capital directly. This is only true indirectly. In all but a few exceptional cases, new stock issues are allocated to reputation makers who use their own reputations to resell the shares at higher prices in short order. Similarly, there are smaller outfits, such as penny stock promoters, who use their reputations, or in some cases, their lack of reputation, to sell a variety of worthless securities.
Another problem is the general paucity of information. Many market theorists believe that the the market is perfectly efficient in the way it uses information. In fact, the market information the market deals with is reputation, not any serious economic or financial data. The raw information, which may be of dubious quality to begin with, has to be filtered by and then absorbed by investors to achieve a concensus. This means that price movements are often exagerated, and it is often possible to make money on the “dead cat bounce” which is simply an artifact of overdamping in the system. Similarly, the price earnings ratios that are considered outrageous or good deals depend on the reputation of the relevant company, industry sector, and market zeitgeist.
Just as unscrupulous sellers and buyers can manipulate the eBay system, it is also possible for unscrupulous corporate managers, stock brokers, analysts and the like to manipulate the stock exchange pricing system. eBay has the advantage that one can often protect oneself using one’s credit card company as an intermediary. eBay too can be a useful intermediary in that eBay itself has a reputation to maintain. Of course eBay is a much smaller reputation based market place, so the impact of fraud rarely has macroeconomic effect.
@ Dan Simon:
‘We generally don’t have to do that sort of thing anymore, thanks to institutions such as credit card consortia, banking networks and credit reporting agencies. These act as trusted centralized intermediaries, providing identity and reliability information and sometimes transaction indemnity or other guarantees.’
Yes — real-world reputation-tracking companies like Dun and Bradstreet (who obsoleted the “letters of introduction” model) are essential to modern commerce. Similarly, anti-spam reputation companies like Spamhaus perform a similar job online. That’s not at issue, and I agree it’s very much necessary and I can’t see their jobs diminishing.
What ekr is talking about here (I think) is automated reputation systems, trust networks and the like. The “friend of a friend” LinkedIn trust network has really got more in common with those letters of introduction than with D&B…
LOL – reputation is on the table for being solved after a meta identity model!
In fifteenth-century Europe, if you wanted to do business with someone, you brought a sealed letter of recommendation from your cousin, a renowned merchant in Antwerp, and one from your good friend, a Burgher in Frankfurt, and that person would examine the seals and letters carefully, inquire among his network of associates regarding their trustworthiness, and ultimately decide whether your reputation was up to snuff.
We generally don’t have to do that sort of thing anymore, thanks to institutions such as credit card consortia, banking networks and credit reporting agencies. These act as trusted centralized intermediaries, providing identity and reliability information and sometimes transaction indemnity or other guarantees.
Why some people think we should now abandon these modern systems and use the Internet to revive clumsy five-hundred-year-old business practices is just plain baffling to me.
Ed’s points are particularly interesting in light of DIGG now being used in the context of government elections – basically using a reputation system as a polling system with much bigger samples the Gallup and (supposedly but not really) therefore better. If this snowballs, it could get very interesting in 2008!
Are trust networks, where some indication of trust of an unknown party propagates through links to people you actually trust, the same as reputation systems?
What is the current state of trust networks?
Privacy advocates be warned.
The problem is made worse, since each reputation system lives in its own closed data set. eBay’s reputation system looks only at eBay data. Amazon’s looks only at Amazon data.
If they could instead query an aggregate that also included your google search history, flickr/kodak photos, and email history they could be much more confident of your ability to pay. While not a 100% solution, this electronic credit history would surely be able to reduce fraud. That is a very significant and measurable cost.
Single sign-on systems such OpenID seem inevitable and desirable, but will also make this sort of data aggregation even easier.
Perhaps the real problem is that everyone has an equal say in the reputations, even though they haven’t proven themselves to be trustworthy. Or that trust is subjective by nature, so not everyone trusts the same things.
There’s some discussion about this type of stuff in the WOT blog; they are building a different kind of reputation system that’s supposedly more difficult to manipulate, or at least you should be able to tell how reliable the reputations are:
http://www.mywot.com/en/wot/blog/technology/
Lots of information on detecting and/or mitigating such attacks:
http://maya.cs.depaul.edu/~mobasher/cgi-bin/view-pubs.pl?CID=SP
There’s another way to game a system like eBay: set up an account and buy from a seller with a perfect rating and moderate sales. Now be a complete jerk and threaten to leave -ve feedback. Many sellers will go to extreme lengths to protect their 100% +ve, even so far as to take a complete loss on the sale and leave +ve feedback for the buyer. Repeat this process until you have 10 or so +ve feedbacks, which will make you look good to anyone. You are now the buyer from hell but there’s no indication to anyone that this is the case. The system has failed to protect future sellers from an abusive customer.
J
Yeah. I recall this point going around the old cypherpunks lists a decade ago, when reputations systems were going to save the world.
The problem is that the critcs basically don’t get heard compared to the hypesters, since the media has an incentive to report the hype and not the reality. And then some academic wants to jump on the bandwagon for the attention, and the cycle starts all over again.
See this paper on combating fraud on eBay:
“Anomaly Detection as a Reputation System for Online Auctioning”
Shai Rubin, Mihai Christodorescu, Vinod Ganapathy, Jonathon Giffin, Nicholas Kidd, Louis Kruger, and Hao Wang
12th ACM Conference on Computer and Communications Security (CCS 2005)
http://www.cs.wisc.edu/~shai/p12-rubin.pdf
I have seen this in action. E-bay sellers with 50-100 positives, each one a purchase of some electronic text file for 1 cent.
One nice thing about e-bay is that you can see what the transaction was, not just the endorsement. Since e-bay makes more money on more expensive items, a lot of cheap transactions should raise alarm bells.
The closest thing to a general theory of this is Avner Greif’s recent book on Institutions and the Path to the Modern Economy.
Thanks for writing this — I’ll be pointing to this URL repeatedly in the future, since every few months, it’s guaranteed that one of the blogerati will confidently assert that “trust networks / reputation systems are the solution to spam”.
It’d be interesting to get some figures on _exactly_ how much effort and money eBay et al expend on manual reputation management, although I guess that data is very unlikely to see the light of day, given how sensitive they are to fraud issues.
D’oh! The BoingBoing article points to that one, BTW.
AAAAA++++ GREAT INSIGHT! WOULD READ FROM THIS BLOG AGAIN!
Just today there was a great article about buying votes on Digg. Despite what Digg says, it was very easy to game the system.
http://www.wired.com/news/technology/internet/0,72832-0.html