Policy-makers often strive to solicit ideas from the public, but making this process effective is very, very hard. Americans need only think back to the health-reform town hall meetings last summer, many of which devolved into chaos. Online approaches, such as President Obama’s virtual town hall meeting, have also ran into problems. So, how can policy-makers discover the best ideas from the public?
The Organization for Economic Cooperation and Development (OECD)–an intergovernmental think tank–recently faced this challenge. They were planning a global summit for education leaders that is taking place in Paris today (November 4th), and they wanted to bring fresh thinking from the public to this group. To achieve this goal, the OECD created an “idea marketplace” at www.allourideas.org, a website that I have been developing with a team of students and web-developers. An idea marketplace allows groups to collect and prioritize ideas in a way that is democratic, transparent, and open. It’s certainly a lot easier than inviting everyone to Paris for the meeting.
Here’s how it worked for the OECD, more specifically Joanne Caddy, Julie Harris, and Cassandra Davis. They “seeded” the website with about 50 ideas (e.g., “Give teachers regular sabbaticals so that they can be trained outside of regular school time or during holidays.”). Then visitors to their idea marketplace were presented with the question “Which is the more important action we need to take in education today?” and two ideas from the pool. The visitors voted for one of the ideas, and then another pair of ideas was presented. This process of pairwise voting continued for as the long as the visitor wished. Also, at any time the visitor could upload an idea which would then go into the pool of ideas to be voted on by others. In this way the idea marketplace allowed the OECD to collect ideas from the community and have the community prioritize them. Both of these steps–collection and prioritization–are needed for a successful “crowdsourcing” of ideas. Here is what the voting looked like to the visitor:
So, how did it work in practice? In just one month, people from more than 90 countries collectively cast more than 27,000 votes.
Equally impressive, these participants uploaded 325 new ideas, again from all over the world.
By combining the voting process with the open uploading of new ideas, this community of stakeholders was able to generate a ranked list that more truly reflected the opinions of the group. The top five ideas (shown below) will be presented to the education leaders today at the summit in Paris, and critically four of these five ideas were uploaded by visitors. In other words, these were ideas that the OECD did not include on their initial list. This example nicely shows how idea marketplaces enable new ideas to bubble-up, allowing groups to learn about things they didn’t even know to ask about.
The top five ideas were:
- Teach to think, not to regurgitate.
- Commit to education as a public good and a public responsibility.
- Focus more on creating a long-term love of learning and the ability to think critically than teaching to standardised tests.
- Ensure all children have the opportunity to discover their natural abilities and develop them.
- Ensure that children from disadvantaged background and migrant families have the same opportunity to quality education as others.
In addition to the OECD, more than 600 idea marketplaces have been created for free at allourideas.org. These have collected about 250,000 votes and 5,000 uploaded ideas. Other policy-related uses have come from student governments (for example, at Princeton and Columbia) and the New York City Government, which is currently using allourideas.org for two projects:
- The Department of Parks and Recreation is using allourideas.org to prioritize residents’ ideas for the new master plan for the parks in Northern Manhattan.
- The New York City Mayor’s Office of Long-Term Planning and Sustainability is using allourideas.org to integrate residents’ ideas into PlaNYC 2030, New York’s citywide sustainability initiative.
allourideas.org is an open-source research project that has received generous funding from CITP and Google. You can keep up on the project by reading our blog or following us on Twitter or Facebook.
While I agree with the sentiments expressed above, that the suggestions are aspirational, this is only a starting point.
In the 1967 Plowden Report (UK Primary Education) a survey of schools was undertaken to establish those that were meeting criteria thought to be critical to an excellent education.
It’s fairly certain that some schools are turning out disproportionate numbers of children with a life long love of learning, who think critically and go on to lead fulfilling lives, presumable as a result of having their particular “talents” nurtured. It can’t be beyond the wisdom of the OECD to identify which schools are responsible for children who turn into adults with those characteristics.
A good start would be to stop teaching to the exam.
We’ve seen in so many other crowdsourcing venues that things work best when most of the people participating don’t have a lot of resources available to advance their particular view. Once is becomes worth someone’s while to subvert such a forum, it’s a pretty easy task to accomplish.
Once is becomes worth someone’s while to subvert such a forum, it’s a pretty easy task to accomplish.
The pairwise voting as part of the process will slow things down compared to simple ranking schemes. It reminds me a little of the fix that was applied to the Clipper chip way back when, namely to make each encryption call take something like a second. In the noise to someone using it the way it was supposed to be used, but a noticeable delay to people trying to subvert it.
I have to say, the final ideas they came up with do not seem encouraging for this approach. The final ideas are… well, lame. They are aspirational. They are motherhood and apple pie, with no specificity.
Sure, probably everyone agrees all five of those “ideas” would be great. But they’re not an idea for how to improve education; they’re simply a fantasy wishlist. The hard part is not getting agreement that these would be a great outcome; the hard part is figuring out how to achieve them, given the constraints we are under. What will change to make it possible to achieve these goals in the future, when we haven’t been able to do so up till now? What new methods would improve our ability to achieve these goals? What new resources will be made available? What are we prepared to sacrifice to achieve these goals?
Overall, this experiment sounds like a failure: a feel-good project that delivered nothing of true value.
Hi Critic,
The ideas are limited to 140 characters so I don’t think we should expect such rich results.
Also, if you look at all the ideas, not just the top 5, you would see that most of them aspirational things that most people would agree with. What I think this system does is provide a democratic and transparent process that allows large groups of people to prioritize these ideas. Further, because the system is open to new ideas, people have the chance of learning about things they didn’t know ahead of time. While “Teach to think, not to regurgitate.” might seem obvious after the fact, it was not on the list of 50 ideas that OECD used to seed the process.
So I agree with you that it was a failure if the goal was the revolutionize education across the world (or even more modestly, to develop a plan to revolutionize education across the world). However, if the goal was more modest — have a democratic and open way of bringing new ideas from all over the world to the attention of policy-makers (all at low cost and effort) — then I would say it was a success.
I have encountered the problem that one idea is really bad placed against a really good idea. Wow. What a choice. So “ideas” are rigged to in the beginning. I keep thinking what stupid ideas. They need another choice like both are stupid.
The ideas are not “rigged”; they are paired randomly. If you don’t like either idea you can click “I can’t decide” and then say you don’t like either idea.
This seems like a very interesting system for collecting information from the public and for separating the wheat from the chaff.
I would point out a couple of things that I think should be taken into consideration.
1) On examining the list of five “winning” ideas, it seems to me that some of them are minor variations of each other. The whole approach would seem prone to this sort of thing. People who see a good idea, but think they have an even better twist will upload that new twist, and subsequent voters, when presented with any variant of the good idea will choose it in preference to a slightly less good idea that addresses a different problem. On the flip side, with too many variants on the same thing, it is possible that no individual variant would get enough votes to raise it to the level of importance that it would get if all were treated as a single unit. I suspect that the process could be tweaked to address this issue, but some thought would have to be put into it.
2) When I went to the NYC parks site that you referenced, another issue occurred to me. As a New York ex-pat (i.e. I moved to NJ), I feel it is my birthright to help decide what’s good for NY parks 🙂 However, the website had no way of authenticating me as a New Yorker. Now, I realize that very few people would have much interest in pushing any particular approach to making NYC parks more pleasant. However, in more controversial applications (such as health-care reform, which was the town-hall topic you used to motivate this posting in the first place), I can see interested parties launching a concerted effort to game the process to drive idea selection in their chosen directions.
3) Somewhat related to 2: The sample of people who would participate in an on-line crowd-sourcing idea-generation approach might be a bit biased. It is limited to (a) people who have access to the internet and (2) have heard of the existence of this crowd-sourcing website, which may mean people who have heard about it from some interested party. This will probably be less of an issue over time, as more people get access to the internet and as this sort of approach becomes more mainstream, but publicity methods will always be an issue, I think. Not that the people who show up to a town-hall meeting in person are any less motivated by agendas, but it is harder to get out live bodies than to set out bots to submit and vote on “ideas”.
4) Good ideas that are uploaded late in the voting process are inherently at a disadvantage. There will be fewer people from whom to garner votes. On the flip side, weaker ideas that come in early may seem stronger than they would if they had come in when all ideas are available to vote on.
Again, none of this is fatal…I would just like to know what thought has been given to these issues.
Hi dmc,
These are all interesting issues that you raise, and ones that we have considered. Here are some thoughts about them.
1) slight variations of the same idea
It is true that people can upload slight variations of the same idea. However, assuming that there are a sufficient number of votes to sort through everything, this is not a problem; the system will just find best variation. Because of the pairwise voting, we don’t need to worry about similar ideas “splitting the vote.”
In some other idea marketplaces I have seen very similar ideas (e.g., “smaller student body” and “fewer students”) get uploaded and fortunately they end up with the same score. Thus, one could also think of these “duplications” as “replications” that help validate the voting system.
2) authentication
It is true that the system currently does not attempt to authenticate voters. That was a conscious decision because we wanted to lower barriers to participation as much as possible. Even a simple log-in could deter large numbers of voters, and for the NYC Parks it is not even clear how one would attempt such an authentication. Groups that want authentication could accomplish that task by embedding the idea marketplace within an intranet.
3) differential participation/fraud
Sure, I totally agree that this can only capture ideas from people with internet access and who have heard about the idea marketplace. In terms of internet access, I think that problem will decrease over time (as you suggest). As far as spreading the word about the idea marketplace, many groups that use allourideas have a pre-existing email list of all group members (e.g., Princeton Undergraduate Student Government). By emailing everyone on the list, you can ensure that everyone knows about it. Of course, that does not ensure that everyone will participate.
4) ideas uploaded late in the process
Ideas uploaded late in the process are not at an inherent disadvantage. That is because our scoring system is based not on the number of votes an idea received, but the estimated probability that this idea will beat a randomly chosen idea. Thus, few votes just means we estimate the probability with greater uncertainty. For that reason, our algorithm for choosing which pairs to show tends to show pairs with new ideas more frequently. While there are limits to our current approach (e.g., if there are hundreds of ideas uploaded at the very end it will overwhelm our algorithms), I think we actually do a decent job of dealing with the fact that ideas are uploaded over time. Critically, the system is designed to prevent a cumulative advantage process which leads early ideas to lock-in.
Those 5 ideas sound all very nice, but they all seem to be in terms of end result. Are there ideas for implementing them?