quetzal_rainbow

Wiki Contributions

Comments

I casually thought that Hyperion Cantos were unrealistic because actual misaligned FTL-inventing ASIs would eat humanity without all that galaxy-brained space colonization plans and then I realized that ASI literally discovered God on the side of humanity and literal friendly aliens which, I presume, are necessary conditions for relatively peaceful coexistence of humans and misaligned ASIs.

The main problem here is:

I guess that step 4 is probably incomputable. The human body is far, far too complex to model exactly, and you have to consider the effect of your weapon on every single variation on the human body, including their environment, etc, ensuring 100% success rate on everyone. I would guess that this is too much variation to effectively search through from first principles. 

You don't need to do any fancy computations to kill everyone, if you come so far that you have nanotech. You just use your nanotech to emulate good old biology and synthetize well-known botulotoxin in bloodstream, death rate 100%.

The problem with such definition is that is doesn't tell you much about how to build system with this property. It seems to me that it's a good-old corrigibility problem.

Another Tool AI proposal popped out and I want to ask question: what the hell is "tool", anyway, and how to apply this concept to powerful intelligent system? I understand that calculator is a tool, but in what sense can the process that can come up with idea of calculator from scratch be a "tool"? I think that first immediate reaction to any "Tool AI" proposal should be a question "what is your definition of toolness and can something abiding that definition end acute risk period without risk of turning into agent itself?"

I would say that it should be done using google forms? For usability of large statistics.

The main problem here is "how to elicit simulacra of superhuman aligned intelligence while avoiding Waluigi effect". We don't have aligned superintelligence in training data and any attempts to elicit superintelligence from LLM can be fatal.

How much should we update on current observation about hypothesis "actually, all intelligence is connectionist"? In my opinion, not much. Connectionist approach seems to be easiest, so it shouldn't surprise us that simple hill-climbing algorithm (evolution) and humanity stumbled in it first.

I see some funny pattern in discussion: people argue against doom scenarios implying in their hope scenarios everyone believes in doom scenario. Like, "people will see that model behaves weirdly and shutdown it". But you shutdown model that behaves weirdly (not explicitly harmful) only if you put non-negligible probability on doom scenarios.

I am not a domain expert, but I get the impression that the primary factors of Pareto-frontier for software industry is "consumer expectations" and "money costs", and primary component of money costs is "programmer labor", so software development goes mostly on the way "how to satisfy consumer expectations with minimum possible labor costs", which doesn't put much optimisation pressure on computing efficiency. I frankly expect that if we spend bazillion dollars on optimisation, we can at least halve required computing power for "Witcher 3". Demoscene proves that we can put many things in 64KBytes of space.

Load More