Devaluing proposals only because they purportedly originated with an adversary.
In any technical discussion, there are a lot of well intentioned but otherwise not so well informed people participating.
On the part of individuals and groups both, this has the effect of creating protective layers of isolation and degrees of separation – between the qualified experts and everyone else.
While this is a natural tendency that can have the beneficial effect the creation of too specific or strong of an 'in-crowd' can result in mono-culture effects.
The problem of a very poor signal to noise ratio from messages received from people outside of the established professional group basically means that the risk of discarding a good proposal from anyone regarded as an outsider is especially likely.
In terms of natural social process, there does not seem to be any available factor to counteract the possibility of forever increasing brittleness in the form of decreasing numbers of new ideas (ie; 'echo chambers').
- link Wikipedia: Reactive devaluation
- an item on Forrest Landry's compiled list of biases in evaluating extinction risks.
if the proof is to be formal, it needs to plug into the mathematical formalizations one would use to do the social science form of this. and if you want to deal with certain kinds of social systems, it's more complicated than that. I'm also an experienced generalist nerd with a lot to say; unfortunately, even if we're real good at babbling, they're right to look at people funny even if they have the systems programming experience or what have you. seems like you're doing stuff that doesn't plug into reality very well. can you provide a demonstration of it as a multi-agent simulation, for example? various levels of shareable multi-agent sim tool to demonstrate whatever effect you'd like; then the only proof you need provide circumstantially is the applicability of the simulation, which means if we find a way the metaphor breaks, we can go back to the drawing board for the simulated example; in software and life in general, tests only give us coverage of the example we try, but at least we can assert that the test passes.
Could you clarify what you mean with a "social science form" of a mathematical formalisation?
I'm not familiar with this.
It was expected and understandable that people look funny at the writings from a multi-skilled researcher with new ideas that those people were not yet familiar with.
Let's move on from first impressions.
If with simulation, we can refer to a model that is computed to estimate a factor on which further logical deduction steps are based on, that would connect up with Forrest's work (it's not really about multi-agent simulation though).
Based on what I learned from Forrest, we need to distinguish the 'estimation' factors from the 'logical entailment' factors. That the notion of "proof" is only with respect to that which can be logically entailed. Everything else is about assessment. In each case, we need to be sure we are doing the modelling correctly.
For example, it could be argued that step 'b' below is about logical entailment, though according to Forrest most would argue that it is an assessment. Given that it depends on both physics and logic (via comp-sci modeling), it depends on how one regards the notion of 'observation', and where that is empirical or analytic observation.
Note again: the above is not formal reasoning. It is a super-short description of what two formal reasoning steps would cover.
but if we can take the type signature from a simulation, then we can attempt to do formal reasoning about its possibility space given the concrete example. if we don't have precise types, we can't reason through these systems. b seems to me to be a falsifiable claim that cannot be determined true or false from pure rational computation, it requires active investigation. we have evidence of it, but that evidence needs to be cited.
How does your approach compare with https://www.metaethical.ai/?
This insight feels relevant to a comment exchange I was in yesterday. An AI Safety insider (Christiano) lightly read an overview of work by an outsider (Landry). The insider then judged the work to be "crankery", in effect acting as a protecting barrier against other insiders having to consider the new ideas.
The sticking point was the claim "It is 100% possible to know that X is 100% impossible", where X is a perpetual motion machine or a 'perpetual general benefit machine' (ie. long-term safe and beneficial AGI).
The insider believed this was an exaggerated claim, which meant we first needed to clarify epistemics and social heuristics, rather than the substantive argument form. The reactions by the busy "expert" insider, who had elected to judge the formal reasoning, led to us losing trust that they would proceed in a patient and discerning manner.
There was simply not enough common background and shared conceptual language for the insider to accurately interpret the outsider's writings ("very poor signal to noise ratio from messages received").
Add to that:
I mean, someone recognised as an expert in AI Safety could consciously mean well trying to judge an outsider's work accurately – in the time they have. But that's a lot of biases to counteract.
Forrest actually clarified the claim further to me by message:
But the point is – few readers will seriously consider this message.
That's my experience, sadly.
The common reaction I noticed too from talking with others in AI Safety is that they immediately devaluated that extreme-sounding conclusion that is based on the research of an outsider. A conclusion that goes against their prior beliefs, and against their role in the community.
The tricky thing here is that a few people are reacting by misinterpreting the basic form of the formal reasoning at the onset, and judging the merit of the work by their subjective social heuristics.
Which does not lend me (nor Forrest) confidence that those people would do a careful job at checking the term definitions and reasoning steps – particularly if written in precise analytic language that is unlike the mathematical notation they’re used to.
The filter goes both ways.
Actually, this post was written in 2015 and I planned last week to reformat it and post it. Rereading it, I’m just surprised how well it appears to line up with the reactions.
Really appreciate you sharing your honest thoughts her, Rekrul.
From my side, I’d value actually discussing the reasoning forms and steps we already started to outline on the forum. For example, the relevance of intrinsic vs extrinsic selection and correction, or the relevance of the organic vs. artificial substrate distinction. These distinctions are something I would love to openly chat about with you (not the formal reasoning – I’m the bridge-builder, Forrest is the theorist).
That might feel unsatisfactory – in the sense of “why don’t you just give us the proof now?”
As far as I can tell (Forrest can correct me later), there are at least two key reasons:
There is a tendency amongst AI Safety researchers to want to cut to the chase to judging the believability of the conclusion itself. For example, notice that I tried to clarify several argument parts in comment exchanges with Paul, with little or no response. People tend to believe that this would be the same as judging a maths proof over idealised deterministic and countable spaces. Yet formal reasoning here would have have to reference and build up premises from physical theory in indeterministic settings. So we actually need to clarify how a different form of formal reasoning is required here, that does not look like what would be required for P=NP. Patience is needed on the side of our interlocutors.
While Forrest does have most of the argument parts formalised, his use of precise analytical language and premises are not going to be clear to you. Mathematicians are not the only people who use formal language and reasoning steps to prove impossibilities by contradiction. Some analytical philosophers do too (as do formal verification researchers in industrial software engineering using different notation for logic transformation, etc.). No amount of “just give the proof to us and leave it to us to judge” lends us confidence that the judging would track the reasoning steps – if those people already did not track correspondences of some first basic argument parts described by the explanatory writings by Forrest or I that their comments referred to. Even if they are an accomplished mathematician, they are not going to grasp the argumentation if they skim through the text, judging it based on their preconception of what language the terms should be described in or how the formal reasoning should be structured.
I get that people are busy, but this is how it is. We are actually putting a lot of effort and time into communication (and are very happy to get your feedback on that!). And to make this work, they (or others) will need to put in commensurate effort on their end. It is up to them to show that they are not making inconsistent jumps in reasoning there, or talking in terms of their intuitive probability predictions about the believability of the end result, where we should be talking about binary logic transformations.
And actually, such nitty-gritty conversations would be really helpful for us too! Here is what I wrote before in response to another person’s question whether a public proof is available:
Also note that Forrest’s formal reasoning work got funded by a $170K grant by Survival and Flourishing Fund. So some grant investigators were willing to bet on this work with money.
I respect this take then by Paul a lot. This is how I also started to think about it a year ago.
BTW, I prefer you being blunt, so glad you’re doing that.
A little more effort to try to understand where we could be coming from would be appreciated. Particularly given what’s at stake here – a full extinction event.
Neither Forrest nor I have any motivation to post unsubstantiated claims. Forrest because frankly, he does not care one bit about being recognised by this community – he just wants to find individuals who actually care enough to consider the arguments rigorously. Me because all I’d be doing is putting my career at risk.