So, what exactly is the difference between Disputation Arenas and medieval scholastic disputations (besides that the former is on the Internet)?
Hm. I don't see any "hold off on proposing solutions" methods. Though I guess prediction markets can be solution-agnostic.
Anyhow, that (plus background knowledge) makes me think there's a lot of room for improvement here, but that discriminating between possible improvements is really hard.
"Hold off on proposing solutions" is an important technique because the Human brain is lazy, and once it thinks of one solution, it will not try to look for another.
I'd say that the interface between the "centrifugal phase" and the "centripetal phase" implicitly reduces the explicit need to protect ideation using "hold off on proposing solutions" - sure, you can present the solution you thought about in the "centrifugal phase" immediately, but the solution gets pushed into the meat grinder of whatever "centripetal phase" there is, as it must compete against other solutions. Ideally, none of the solutions presented at the start of the centripetal phase will be designated as the "best" solution (hopefully, given the anonymizing effects of Delphi and the self-consistency pushed on you by writing your ideas in the NGT (nominal group technique)).
Even in brainstorm sessions, "hold off on proposing solutions" is needed only if the initial idea(s) presented are given undue weight compared to later ideas. Delphi causes the initial ideas to be mixed with the others - ideally, your summarizer will be given the expert's answer sheets in random order, and in the real-time online form that's the reason why the group qualitative answer is randomized. Ideally in an NGT the facilitator will steer everyone away from overly discussing one idea at the expense of the rest - it is noted there with an IMPORTANT scare tag, after all. For prediction markets, you don't discuss ideas anyway, so that is not even an issue.
"hold off on proposing solutions" is needed only if the initial idea(s) presented are given undue weight compared to later ideas.
Hmm. Why do you think that? As the number of potential solutions increases, I feel like even proposing solutions gets harmful.
Because the article about it specifically mentions that this is the failure mode to avoid:
Norman R. F. Maier noted that when a group faces a problem, the natural tendency of its members is to propose possible solutions as they begin to discuss the problem. Consequently, the group interaction focuses on the merits and problems of the proposed solutions, people become emotionally attached to the ones they have suggested, and superior solutions are not suggested. Maier enacted an edict to enhance group problem solving: "Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any."
So "hold off on proposing solutions" is just one possible solution. Deciding to take that solution immediately, without considering other options (such as NGT's approach) is precisely falling into that same trap.
In short, hold off on proposing the solution of "hold off on proposing solutions". v(^.^)v
edit:
Consider that under NGT, you are given 10 to 15 minutes to think of solutions before anyone gets to propose any solutions. That strikes me as longer than a typical "hold off".
Somehow I think that "let's follow best practices in our cognition" isn't exactly a 'proposed solution' in the sense that one should be holding off from doing it.
shrug it's best practice at a particular time and place, but is it the best practice at all times and places?
I'll grant that the procedure "tell all participants: 'hold off on proposing solutions'" is a good procedure in general, but is it the best procedure under all circumstances? How about enforcing the "hold off" part, rather than just saying it to participants? (cref. NGT's silent idea generation).
is precisely falling into that same trap.
Well, close. But as life is not always that cute, not quite :P
Consider that under NGT, you are given 10 to 15 minutes to think of solutions before anyone gets to propose any solutions. That strikes me as longer than a typical "hold off".
I agree that this will definitely help with the social bits - I'm worried about the bits that are internal to a person, where people just have some common failure modes when trying to solve problems. To give a personal anecdote, say I give someone a polarizing filter, a light intensity sensor, and a piece of plastic and say "when this piece of plastic is at an angle, what happens to the light?" If they immediately start looking for solutions, rather than playing around, they will fail. 9 times out of ten. Kiss of death, no social stuff needed. The people who figure out the correct answer with any reasonable rate are the same people who explore things just to explore things.
I'm worried about the bits that are internal to a person, where people just have some common failure modes when trying to solve problems.
shrugs Well, seatbelts don't stop accidents, but they do reduce the side effects of getting into one. While the disputation arenas do not directly prevent such internal failure modes, they help prevent that internal failure mode in a key influential person from spreading to the rest of the group. Yes, hold off on proposing solutions (don't drink and drive). But also put some extra railing and padding so that others making a mistake do not necessarily get you into error either (seatbelts)
Why don't you discuss the status quo solution? LessWrong is a system for allowing rationalists to think together. It's not highly formalized but that makes it a lot more flexible.
If you say you want groups of rationalists to solve problems together, which problems are you thinking about? What sort of problems do you want to solve?
LW is a venue for rationalists to think together, but not a technique for rationalists to think together. Despite the term "disputation arenas", what almkglor is talking about is the latter, not the former.
I suppose you could say that "start a discussion in LW and see what happens" is a "disputation arena" in almkglor's sense. So, therefore, is "just get a bunch of people together in a room and let them talk about it". Presumably the techniques almkglor describes were designed because just putting people together in a room has been found not to work very well. Do you have grounds for thinking that putting people together in an LW thread works better? Or that rationalists are immune to groupthink and failure to reach consensus?
Or that rationalists are immune to groupthink and failure to reach consensus?
I think that those two criteria's are insufficent to judge the quality of a "disputation arena" for rationalists. The core problem is that encouraging participation isn't one of the criteria. If you want to get things done in the real world than it's vitally important to encourage participation. A disputation arena without participants is worthless.
I also doubt that reaching consensus is always a good thing. Singularity is one of the topic that almkglor thinks about. If you would take a year to get all LessWrong participants to have a consensus belief about the singularity I think that would be bad.
In year two you will have massive group think problems when you continue to discuss the singularity because all participants know the consensus belief of year one. I would prefer a system with more diversity in opinions.
As far as avoiding group think, there are other strategies. Encouraging more members of the community to play devil's advocate would be one way.
The core problem is that encouraging participation isn't one of the criteria.
I think you may have been led astray by the terminology into thinking that "disputation arena" means, y'know, an arena for disputation, when in fact it seems to mean a technique for discussing things. Techniques like the Delphi method are intended for groups that already exist and need to do some thinking.
I also doubt that reaching consensus is always a good thing.
Is anyone claiming it is? My understanding is that these "disputation arenas" are methods a group can use to arrive at consensus when they need to do so. (Also #1: I'd think most of them are adaptable to the case where you don't particularly need a consensus as such. Also #2: a consensus can be a complicated one with probabilities and things in, and it seems to me that agreement on such a consensus would avoid many of the perils of the usual sort of groupthink.)
I prefer "disputation arena" because "group thinking" is too close to "groupthinking".
Is there a better term for "techniques for discussing things so that lots of thinking people can give their input and get a single coherent set of probabilities for what are the best possible choices for action" other than "disputation arena" or "group thinking technique"?
I do want to be precise, and "disputation arena" sounded kewl, but whatever.
I don't know of any other term with that meaning. Making one up wouldn't really be any worse than using "disputation arena", I think, because to an excellent first approximation no one knows what "disputation arena" means anyway.
Techniques like the Delphi method are intended for groups that already exist and need to do some thinking.
I don't think that's the goal layed out in the first paragarph of the post. It ends with:
This makes it not only desirable to find ways to effectively get groups of rationalists to think together, but also increasingly necessary.
Getting groups of rationalists to think together is a goal where it's important to design the system in a way that makes it easy and motivates participants to participate.
Okay, so that's a sub-goal that I didn't think about. I will think about this a little more.
Still, assuming that group exists and needs to do some thinking together, I think techniques like Delphi are fine.
Anyway, I assumed that LW's groups are more cohesive and willing to cooperate in thinking exercises in groups (this is what I was thinking when I said "This makes it not only desirable to find ways to effectively get groups of rationalists to think together, but also increasingly necessary."), but apparently it's not as cohesive as I thought.
Successful online communities have a low bar to entry. As a result they aren't as cohesize as a hierachical institution where you can simply order a group to make some decision via Delphi.
LessWrong is a network. It's no hierachical institution and isn't market driven.
If you want some high level understanding of the network paradigma, I recommend "In Search of How Societies Work" by David Ronfeld.
LessWrong is one way of implementing groups of rationalists thinking together. One might say that it provides a centripetal phase: the discussion forums. But what centrifugal phase exists that prevents groupthink? Yes, we have "hold off on proposing solutions" - but remember that no current rationalist is perfect, and LW may grow soon (indeed, spreading rationality may require growing LW).
Also remember that people - including LessWrong members - tend to favor status quos, and given a chance, people tend to defend status quos to the death.
At the very least, we need to consider what other systems are available, and specifically de-emphasize the local status quo, since we might not be thinking perfectly rationally about it.
It's not highly formalized but that makes it a lot more flexible.
The Turing machine is highly formalized and is the most flexible possible computational machine. I get "false dichotomy" signals from this statement.
If you say you want groups of rationalists to solve problems together, which problems are you thinking about? What sort of problems do you want to solve?
insane governments, insane societies, insane individuals, and the singularity, in that rough order of priority.
The Turing machine is highly formalized and is the most flexible possible computational machine. I get "false dichotomy" signals from this statement.
I don't think you understand what I mean with the word highly formalized in this context. LessWrong has also a bunch of rules. Those rules are however made in a way where they don't constrain the way one can use LessWrong as much as the rules of Delphi constrain it's participants.
At the very least, we need to consider what other systems are available, and specifically de-emphasize the local status quo, since we might not be thinking perfectly rationally about it.
No, if you propose an alternative it makes sense to explain how it would improve the status quo. Ignoring the status quo that provides a system that actually works in practice is a bad idea.
At the moment there no working Delphi system that allows rationalists to discuss solutions for handling insane governments. The cases where Delphi was used successfully are cases where it gets implented top-down. Whether the same approach works in an online community is up for discussion. I don't know of a single case where such a system got enough users to work.
insane governments, insane societies, insane individuals, and the singularity, in that rough order of priority. InTrade style prediction markets have the issue that predictions need to be able to be judged as true or false within a reasonable timeframe.
If you want to discuss how to tackle "insane governments" restristing yourself to claims that can be judged as true or false in short time frames probably removes most of the interesting questions from the discussion.
If you think otherwise, please illustrate how you would tackle the issue you brought forward in your post with Prediction Markets. How to tackle it with Delphi would also be interesting.
I'm also not clear about why we need to find consensus on "insane governments, insane societies, insane individuals, and the singularity".
I don't think you understand what I mean with the word highly formalized in this context. LessWrong has also a bunch of rules. Those rules are however made in a way where they don't constrain the way one can use LessWrong as much as the rules of Delphi constrain it's participants.
Okay, what exactly do you mean by "highly formalized"?
Constraints on behavior are not necessarily bad, in much the same way that there are more things in heaven and earth than are dreamt of in our philosophy: constraining things to a subset that can be shown to work can help. So I don't really see "current LW has more freedom!!" as a significant advantage - because it might have more freedom to err. Of course, the probability of that being true is low - but can we at least try to show that?
After all, LW code is derived from Reddit. Of course, the online system is just part of the overarching system, and the system as a whole (including current community members) is different (there are more stringent rules for acceptance into the community here than on Reddit), but it might do well to consider that things may be made better.
At the very least, we need to consider what other systems are available, and specifically de-emphasize the local status quo, since we might not be thinking perfectly rationally about it.
No, if you propose an alternative it makes sense to explain how it would improve the status quo. Ignoring the status quo that provides a system that actually works in practice is a bad idea.
I said "de-emphasize", not ignore. What I mean by "de-emphasize" is, acknowledge its existence, but treat it as an idea you have already thought about, i.e. keep it on hand and don't forget about it, but don't keep thinking about it at the expense of other, external ideas. In any case, I thought that it would be unnecessary to have to discuss the local status quo, since I would assume that members already know it.
Should I discuss the current status quo? I am not a regular member, despite reading OB before and LW for years, so I don't feel qualified to get into its details. I mostly read the sequences and hardly look at discussion. Or even comments on the articles, anyway. So my knowledge of LW informal rules are minimal to say the least. Can you describe the status quo for me?
At the moment there no working Delphi system that allows rationalists to discuss solutions for handling insane governments. The cases where Delphi was used successfully are cases where it gets implented top-down. Whether the same approach works in an online community is up for discussion. I don't know of a single case where such a system got enough users to work.
So should we, at this point, completely discard Delphi methods? How about NGT?
I suspect that it's possible to modify LW's polls to add some kind of Real-Time Delphi Method, as I mentioned in the article: (1) allow members to change their chosen options (2) require members to give a short justification for their chosen option (3) give randomized samples of justifications from other members. We can even have a flag that specifies normal forum polls or Delphi-style polls. But if the cost of making this modification is higher than the expected probability of that kind of Delphi being successful times the expected utility of that kind of Delphi methods in general for the rest of LW's lifetime, then fine - let's not do it.
If you think otherwise, please illustrate how you would tackle the issue you brought forward in your post with Prediction Markets. How to tackle it with Delphi would also be interesting.
I don't know how to tackle it with Prediction Markets other than by futarchy: first vote on what measurements are to be used, then run a prediction market about whether particular policy decisions will improve or reduce those measurements. Insane governments are more sane if they have less corruption, better bureacratic efficiency blah blah - we may need to vote on that. Then we need to propose actual policy decisions and predict if they will lead to less corruption etc. or not. Unfortunately, I don't understand enough of futarchy yet to make a proper judgment about it - it's currently a mostly black box to me. I'm disturbed that futarchy_discuss appears to be defunct - I'm not sure if it's because prediction markets have turned out to fail badly, or what.
Assuming those same measures can be agreed upon - less corruption, better bureacratic efficiency - then I suppose a Delphi Method can be made with "what policies should reduce corruption blah blah? How can we impose those policies from below? What feasible actions can we use to get those policies accepted?" as the questions.
(if you think that my definition of "insane government" isn't very good, please understand that I live in a shitty little third-world country where the most troubling problems of the government is corruption and inefficiency, not whether or not the government should raise taxes)
I'm also not clear about why we need to find consensus on "insane governments, insane societies, insane individuals, and the singularity".
Because I think lack of consensus is one reason why our kind can't cooperate.
Can we at least try to pull together on this one?
I said "de-emphasize", not ignore.
You did write a long post on different systems for discussion and you did ignore it in that post.
Within your list you didn't discuss systems that have shown to work in the real world to solve the kind of issues that you want to solve. If you don't like LessWrong as an example take an online community like Wikipedia as an example. If you don't know the specifcs of any system that actually works in the real world, you are in a poor position to propose new system.
(if you think that my definition of "insane government" isn't very good, please understand that I live in a shitty little third-world country where the most troubling problems of the government is corruption and inefficiency, not whether or not the government should raise taxes)
I would say that in the US corruption and government ineffiency are also central problems.
If you however want to solve those kinds of problems in your country than you have to choose. One way would be to get the IWF to promote some Good Government program in your country in a top-down way. The other way involves finding supporters in your own country.
For both strategies I doubt that the LessWrong public is the right audience. Join/found some Liquid Feedback based political party in your country.
You might even try to adept Liquid Feedback to be more Delphi like.
Because I think lack of consensus is one reason why our kind can't cooperate.
One of the most effective calls for support to highly intelligent nerds was probably Julian Assange's call that among other thing involved him telling the audience that they won't get Christmas presents when they don't cooperate. Julian Assange didn't try to organise some vote to get consenus.
You did write a long post on different systems for discussion and you did ignore it in that post.
I thought it would be unnecessary, as I thought the people here would already know and it would be repetitive to do reiterate what is already known here. I'll try to see if I can come up with some description of the local status quo, then, and edit the article to include it. I'm a little busy, Christmas is important in this country.
Within your list you didn't discuss systems that have shown to work in the real world to solve the kind of issues that you want to solve.
Huh? These are techniques that have been studied with papers backing them (at least according to some very basic searches through Google). I have no idea how good those papers are, but maybe you do. Can you show some study specifically showing that Delphi works worse then typical internet forums?
take an online community like Wikipedia as an example.
Again, since LW also has a Wiki, I thought it would be superfluous to add it to the article too. I'll find time to update it then.
If you however want to solve those kinds of problems in your country than you have to choose. One way would be to get the IWF to promote some Good Government program in your country in a top-down way. The other way involves finding supporters in your own country.
For both strategies I doubt that the LessWrong public is the right audience. Join/found some Liquid Feedback based political party in your country.
Thank you for this information.
One of the most effective calls for support to highly intelligent nerds was probably Julian Assange's call that among other thing involved him telling the audience that they won't get Christmas presents when they don't cooperate. Julian Assange didn't try to organise some vote to get consenus.
Okay.
Consider the following statement: two heads are better than one.
It seems obvious to me that several rationalists working together can, effectively, bring more precious brainpower to bear on a problem, than one rationalist working alone (otherwise, what would be the point of having a Less Wrong forum community? You might as well just leave it as a curated community blog and excise the discussion forums.). Further, due to various efforts (HPMOR especially) it appears that LW is inevitably growing. This makes it not only desirable to find ways to effectively get groups of rationalists to think together, but also increasingly necessary.
It is also desirable that methods of getting groups to think should be feasibly doable over the Internet. (I am aware that real-life meetups and stuff exist, but please be reminded that some people in the world do have to live in shitty little third-world countries and might not at all find it economically feasible to go to first-world countries with atrociously high costs-of-living)
So first, let us start with the current "best methods" of getting groups of Traditional Rationalists to coordinate and think, while avoiding groupthink effects that diminish our aggregate rationality. Hopefully, we can then use it as the basis of part of the art of rationalist group thinking. So I'll discuss:
One of my favorite SF authors, David Brin, talks a bit about what he calls "disputation arenas". I won't discuss his ideas here, since his concept of "disputation arena" is actually a relatively "new", raw, and relatively untested procedure - what I intend to discuss for now are things that have at least been studied more rigorously than just a bunch of blog posts (or personal website pages, whatever).
However, I do want to co-opt the term "Disputation Arena" for any process that tries to achieve the following:
We want our group rationality process to avoid groupthink (possibly at some expense of efficiency) because actual, real-world rationalists are not perfect Bayesian reasoners - two words: Robert Aumann. Because rationalists are not perfect, we do not expect a clear consensus to form after the end of the process (i.e. Aumann's does not necessarily apply), so the process we use must force some consensus to become visible.
One thing that David Brin discusses is the general division of the procedures into two "phases":
This seems to me to be a good way of labeling parts of any group-coordination process that attempts to avoid groupthink and achieve consensus.
In my personal research, I've found three things that attempt to achieve those two goals (avoid groupthink, achieve consensus) and which might (perhaps with a stretch) be considered as approximately having two phases (centrifugal and centripetal).
The Delphi Method was originally developed by RAND Corporation (Project RAND at that time, and no relation to Ayn Rand) in order to get better predictions on future military hardware. It is currently used to get better utilization of current human wetware. (^.^)v
Pen and paper version:
Internet version (Wikipedia:Real-time Delphi):
Delphi methods avoid groupthink largely by anonymity: this avoids the bandwagon effect, the halo effect, and the mind-killer. Anonymity and constant feedback also encourage people to revise their positions in light of new information from their peers (by reducing consistency pressure): in non-anonymous face-to-face meetings, people tend to stick to their previously stated opinions, and to conform to the meeting leader(s) or their own bosses in the meeting. A lot of those effects is reduced by anonymity. Pen-and-paper form makes anonymity much easier, since the summary gets the tone and language patterns of the summarizer; some amount of anonymity is lost in the online version (since language patterns might theoretically be analyzed) but hopefully the small sample size (just a short sentence or two) can make language pattern analysis difficult. Note that randomizing the order of the comments in the online version is important, as this reduces the effects of anchoring; sorting by time or karma may increase groupthink due to anchoring on earlier comments, but if each expert sees different "first comments", then this bias gets randomized (hopefully into irrelevancy).
Delphi methods achieve consensus by the summary (which often serves as the "final output" when the process is finished). Arguably, the pen-and-paper version is better at achieving consensus due to the "turn-based" arrival of the summary, which makes the expert pay more attention to the summary, compared to the real-time online system.
The Delphi method's centrifugal phase is the expert's private answering of the questionnaire: each expert makes this decision, and provides the justification, without other's knowledge or help.
The Delphi method's centripetal phase is the act of summarizing, and having the experts read the summary.
I think that forum polls, in general, can be easily adapted into online real-time Delphis by adding the following:
The procedure says "experts" but I think that in something more democratic than the military you're supposed to read that as "anyone who bothers to participate".
Prediction markets are speculation markets built around bets on what things will happen in the future. They are also the core of Robin Hanson's Futarchy, and which you can see somewhere in the background of LW's favorite tentacle alien porn novella (O.o);;.
Pen and paper version:
(what I described above is the simplest and most basic form I found; refer to the Wikipedia article for better elaborations)
Internet version:
Prediction markets avoid groupthink by utilizing the invisible hand. Someone selling you a stock might be an idiot who can't read the tea leaves properly. Or the seller might have knowledge you do not possess, so maybe buying the stock wasn't such a good idea after all? Remember: if you can't find who the sucker on the table is, that sucker is you!! You can't simply assume that what your neighbor says is true and you should sell as many stock of X as possible: maybe he or she is trying to take advantage of you to get your hard-earned cash. Groupthink in such a mistrusting environment gets hard to sustain. Prediction markets work better with very large groups of people, so that you get practical anonymity (although not perfect, in theory you or anyone else can keep track of who's selling to who; online versions are also likely to hide user identities). Anonymity in the prediction market also has the advantages previously described under Delphi Method above.
Prediction markets achieve consensus by utilizing the invisible hand. The price point of any sale serves as an approximate judgment of the epistemic probability of X occurring (or not occurring, depending on the contract that got sold). This gives a real-time signal on what the group of traders as a whole think the probability of X occurring is.
The prediction market's centrifugal phase is each individual trader's thought process as he or she considers whether to buy or sell stock, and at what price.
The prediction market's centripetal phase is any actual sale at any actual price point.
Prediction markets are well-represented online; Intrade is just one of the more famous online prediction markets. Prediction markets appear to be the most popular and widely-known of the disputation arenas I've researched. These all tend to suggest that prediction markets are one of the better disputation arenas - but then remember that the Internet itself has no protection against groupthink.
Nominal group technique is a group decision-making process, appropriate for groups of many different sizes. This procedure's pen-and-paper form is faster than the pen-and-paper forms of the other disputation arenas discussed here.
Pen and paper version:
Unlike the previous procedures, which have been extrapolated into Internet versions, there is currently no Internet version of nominal group technique.
Nominal group techniques avoid groupthink by the two "secret" steps: silent idea generation, and the secret ranking. Having members write down their ideas in the silent idea generation step helps them precommit to those ideas in the idea sharing step, even though more influential group members may present opposite or incompatible ideas. Although the group discussion step disallows explicit criticism of ideas, those criticisms are implicitly expressed during the secret ranking step (i.e. if you have a criticism of an idea, then you should rank it lower).
Nominal group techniques achieve consensus by the idea sharing, group discussion, and ranking steps. In particular, tallying of option rankings is the final consensus-achieving step.
The nominal group technique's centrifugal phase is largely the silent idea generation step, and is the most explicit centrifugal phase among the disputation arenas discussed here.
The nominal group technique's centripetal phase is largely the rest of the procedure.
A modified form of nominal group technique eats up a quarter of my recently-finished novel, Judge on a Boat, which I talked about on LessWrong here, and whose latest raw text source you can read online. Yes, this entire article is just a self-serving advertisement to garner interest in my novel o(^.^o)(o^.^)o.
Compared to the other procedures here, nominal group technique is more complicated and much more dependent on the centralized facilitator; the extreme dependency on the facilitator makes it difficult to create an automated online version. On the other hand, a small group of say 5 to 10 people can finish the nominal group technique in 1 to 2 hours; the other procedures tend to work better when done over several days, and are largely impossible (in pen-and-paper form) to do in a similar time frame. Even the real-time online versions of the other procedures are difficult to do within 2 hours. Prediction markets in particular tend to fail badly if too thin (i.e. not enough participants); for small groups with tight schedules, nominal group technique tends to be the best.
For small groups that need to make a decision within one or two hours, use nominal group technique. It's relatively unwieldy compared to the other disputation arenas, and is less ideal (it has fewer protections against groupthink, in particular), but is fast compared to the others. Also, one might consider parallelizing nominal group technique: split a large group into smaller, randomly-selected sub-groups, have each perform the procedure independently, and then have them send a representative that performs the idea sharing, group discussion, and ranking steps with other representatives (i.e. each sub-group's nominal group technique serves as the silent idea generation of the super-group of representatives). This tends to bias the super-group towards the agendas of the chosen representatives, but if speed is absolutely necessary for a large group, this may be the best you can do.
As mentioned above, prediction markets tend to fail badly if there are too few participants in the speculation market; use it only for extremely large groups that are impossible to coordinate otherwise. In addition, using prediction markets for policy decisions is effectively futarchy; you may want to see the (defunct?) futarchy_discuss Yahoo! group's message archives. In particular the earlier messages in the archive tend to discuss the general principles of prediction markets. Prediction markets are the most famous of the disputation arenas here, but remember that the Internet is not a decent disputation arena.
The Delphi methods seem to be a "dark horse" of sorts. I don't see much discussion online about Delphi methods; I'm not sure whether it's because it's been tried and rejected, or if it simply isn't well known enough to actually be tried by most people. I tend to suspect the latter, since if the universe were in the former case I would at least see some "Delphi Methods suck!!" blog posts.
Both prediction markets and Delphi methods are continuously repeated methods. At any time, the procedure may be stopped or repeated in order to make decisions. However, unlike the nominal group techniques, both are targeted more towards generating advice for decision-makers, rather than making actual decisions themselves.
It may be possible to organize a large, hierarchical group (say a company) with a prediction market for the rank-and-file, some key experts (who should be aware of the prediction market's results) running a Delphi method, and the key decision-making individuals (who read the Delphi method's report) at the top who form a decision using nominal group technique. For more democratic processes, a "poll-style" real-time online Delphi method by itself may work.