All of Florian_Dietz's Comments + Replies

I agree that System 2 is based on System 1 and there is probably no major architectural difference. To me it seems like the most important question is about how the system is trained. Human reasoning does not get trained with a direct input/output mapping most of the time. And when it does, we have to infer what that mapping should be on our own.

Some part of our brain has to translate the spoken words "good job!" into a reward signal, and this mechanism in itself must have been learned at some point. So the process that trains the brain and applies the rew... (read more)

I work in the area of AGI research. I specifically avoid working on practical problems and try to understand why our models work and how to improve them.  While I have much less experience than the top researchers working on practical applications, I believe that my focus on basic research makes me unusually suited for understanding this topic.

I have not been very surprised by the progress of AI systems in recent years. I remember being surprised by AlphaGo, but the surprise was more about the sheer amount of resources put into that. Once I read up on... (read more)

I mean "do something incoherent at any given moment" is also perfectly agent-y behavior. Babies are agents, too.

I think the problem is modelling incoherent AI is even harder than modelling coherent AI, so most alignment researchers just hope that AI researchers will be able to build coherence in before there is a takeoff, so that they can base their own theories on the assumption that the AI is already coherent.

I find that view overly optimistic. I expect that AI is going to remain incoherent until long after it has become superintelligent.

Contemporary AI agents that are based on neural networks are exactly like that. They do stuff they feel compelled to in the moment. If anything, they have less coherence than humans, and no capacity for introspection at all. I doubt that AI will magically go from this current, very sad state to a coherent agent. It might modify itself into being coherent some time after becoming super intelligent, but it won't be coherent out of the box.

This is a great point. I don't expect that the first AGI will be a coherent agent either, though.

As far as I can tell from my research, being a coherent agent is not an intrinsic property you can build into an AI, or at least not if you want it to have a reasonably effective ability to learn. It seems more like being coherent is a property that each agent has to continuously work on.

The reason for this is basically that every time we discover new things about the way reality works, the new knowledge might contradict some of the assumptions on which our goa... (read more)

I agree that current AIs can not introspect. My own research has bled into my believes here. I am actually working on this problem, and I expect that we won't get anything like AGI until we have solved this issue. As far as I can tell, an AI that works properly and has any chance to become an AGI will necessarily have to be able to introspect. Many of the big open problems in the field seem to me like they can't be solved precisely because we haven't figured out how to do this, yet.

The "defined location" point you note is intended to be covered by "being sure about the nature of your reality", but it's much more specific, and you are right that it might be worth considering as a separate point.

Can you give me some examples of those exercises and loopholes you have seen?

A fair point. How about changing the reward then: don't just avoid cheating, but be sure to tell us about any way to cheat that you discover. That way, we get the benefits without the risks.

My definition of cheating for these purposes is essentially "don't do what we don't want you to do, even if we never bothered to tell you so and expected you to notice it on your own". This skill would translate well to real-world domains.

Of course, if the games you are using to teach what cheating is are too simple, then you don't want to use those kinds of games. If neither board games nor simple game theory games are complex enough, then obviously you need to come up with a more complicated kind of game. It seems to me that finding a difficult... (read more)

Yes. I am suggesting to teach AI to identify cheating as a comparatively simple way of making an AI friendly. For what other reason did you think I suggested it?

I am referring to games in the sense of game theory, not actual board games. Chess was just an example. I don't know what you mean by the question about shortcuts.

It needs to learn that from experience, just like humans do. Something that also helps at least for simpler games is to basically provide the manual of the game in a written language.

Is there an effective way for a layman to get serious feedback on scientific theories?

I have a weird theory about physics. I know that my theory will most likely be wrong, but I expect that some of its ideas could be useful and it will be an interesting learning experience even in the worst case. Due to the prevalence of crackpots on the internet, nobody will spare it a glance on physics forums because it is assumed out of hand that I am one of the crazy people (to be fair, the theory does sound pretty unusual).

This solution does not prevent Harry's immediate death, but seems much better than that to me anyway. I haven't been following conversations before, so I can only hope that this is at least somewhat original.

Assumptions:

-Lord Voldemort desires true immortality. Alternatively, there is a non-zero chance that he will come to desire true immortality after a long time of being alive. While he is a sociopath and enjoys killing, achieving immortality is more important to him.

-Lord Voldemort does not dismiss things like the Simulation Hypothesis out of hand. Sinc... (read more)

The nanobots wouldn't have to contain any malicious code themselves. There is no need for the AI to make the nanobots smart. All it needs to do is to build a small loophole into the nanobots that makes them dangerous to humanity. I figure this should be pretty easy to do. The AI had access to medical databases, so it could design the bots to damage the ecosystem by killing some kind of bacteria. We are really bad at identifying things that damage the ecosystem (global warming, rabbits in australia, ...), so I doubt that we would notice.

Once the bots have b... (read more)

I agree. Note though that the beliefs I propose aren't actually false. They are just different from what humans believe, but there is no way to verify which of them is correct.

You are right that it could lead to some strange behavior, given the point of view of a human, who has different priors than the AI. However, that is kind of the point of the theory. After all, the plan is to deliberately induce behaviors that are beneficial to humanity.

The question is: After giving an AI strange beliefgs, would the unexpected effects outweigh the planned effects?

Yes, that's the reason I suggested an infinite regression.

There is also the second reason: it seems more general to assume an infinite regression rather than just one level, since that would put the AI in a unique position. I assume this would actually be harder to codify in axioms than the infinite case.

I know, I read that as well. It was very interesting, but as far as I can recall he only mentions this as interesting trivia. He does not propose to deliberately give an AI strange axioms to get it to believe such a thing.

I do the same. This also works wonderfully for when I find something that would be interesting to read but for which I don't have the time right now. I just put it in that folder and the next day it pops up automatically when I do my daily check.

Can you elaborate on why using dark arts is equivalent ti defecting on the prisoners' dilemma? I'm not sure I understand your line of reasoning.

I'm not entirely sure what you mean by 'Spinoza-style', but I get the gist of it and find this analogy interesting. Could you explain what you mean by Spinoza-style? My knowledge of ancient philosophers is a little rusty.

No, the distinction between MWI and Copenhagen would have actual physical consequences. For instance, if you die in the Copenhagen interpretation, you die in real life. If you die in MWI, there is still a copy of you elsewhere that didn't die. MWI allows for quantum immortality.

The distinction between presentism and eternalism, as far as I can tell, does not imply any difference in the way the world works.

The original distinction. My reconstruction is what I came up with in an attempt to interpret meaning into it.

I agree that my reconstruction is not at all accurate. It's just something that occurred to me while reading it and I found it fascinating enough to write about it. In fact, I even said that in my original post.

The meanings are much clearer now.

However, I still think that it is an argument about semantics and calef's argument still holds.

After reading your comment, I agree that this is probably just a semantic question with no real meaning. This is interesting, because I completely failed to realize this myself and instead constructed an elaborate rationalization for why the distinction exists.

While reading the wikipedia page, I found myself interpreting meaning into these two viewpoints that were probably never intended to be there. I am mentioning this both because I find it interesting that I reinterpreted both theories to be consistent with my own believes without realizing it, and bec... (read more)

I find it surprising to hear this, but it cleans up some confusion for me if it turns out that the major, successful companies in silicon valley do follow the 40 hour week.

That's what I'm asking you!

This isn't my theory. This is a theory that has been around for a hundred years and that practically every industry follows, apparently with great success. From what I have read, the 40 hour work week was not invented by the workers, but by the companies themselves, who realized that working people too hard drives down their output and that 40 hours per week is the sweet spot, according to productivity studies.

Then along comes silicon valley, with a completely different philosophy, and somehow that also works. I have no idea why, and that's what I made this thread to ask.

No, that's not what I mean. The studies I am talking about measure the productivity of the company and are not concerned with what happens to the workers.

I also think that is a possibility, especially the first part, but so far I couldn't find any data to back this up.

As for drugs, I am not certain if boosting performance directly, as these drugs do, also affects the speed with which the brain recuperates from stress, which is the limiting factor in why 40 hour weeks are supposed to be good. I suspect that it will be difficult to find an unbiased study on this.

True, and I suspect that this is the most likely explanation.

However, there is the problem that unless need-for-rest is actually negatively correlated with the type of intelligence that is needed in tech companies, they should still have the same averages over all their workers and therefore also have the same optimum of 40 hours per work, at least on average. Otherwise we would see the same trends in other kinds of industry.

Actually I just noticed that maybe this does happen in other industries as well and is just overreported in tech companies. Does anyone know something about this?

The problem is that during the industrial revolution it also took a long time because people caught on that 40 hours per week were more effective. It is really hard to reliably measure performance in the long term. Managers are discouraged from advocating a 40 hour work week since this flies in the face of the prevailing attitude. If they fail, they will almost definitely be fired since 'more work'->'more productivity' is the common sense answer, whether or not it is true. It would not be worth the risk for any individual manager to try this unless the ... (read more)

I didn't save the links, but you can find plenty of data by just googling something like "40 hour work week studies" or "optimal number of hours to work per week" and browsing the articles and their references.

Though one interesting thing I read that isn't mentioned often is the fact that subjective productivity and objective productivity are not the same.

I think another important point is how simulations are treated ethically. This is currently completely irrelevant since we only have the one level of reality we are aware of, but once AGIs exist, it will become a completely new field of ethics.

• Do simulated people have the same ethical value as real ones?
• When an AGI just thinks about a less sophisticated sophont in detail, can its internal representation of that entity become complex enough to fall under ethical criteria on its own? (this would mean that it would be unethical for an AGI to even think abo

That sounds like it would work pretty well. I'm looking specifically for psychology facts, though.

I am reading textbooks. But that is something you have to make a conscious decision to do. I am looking for something that can replace bad habits. Instead of going to 9gag or tvtropes to kill 5 minutes, I might as well use a website that actually teaches me something, while still being interesting.

The important bit is that the information must be available immediately, without any preceding introductions, so that it is even worth it to visit the site for 30 seconds while you are waiting for something else to finish.

Mindhacks looks interesting and I will keep it in mind, so thanks for that suggestion. Unfortunately, it doesn't fit the role I had in mind because the articles are not concise enough for what I need.

I have started steering my daydreaming in constructive directions. I look for ways that whatever I am working on could be used to solve problems in whatever fiction is currently on my mind. I can then use the motivation from the fictional daydream to power the concentration on the work. This isn't working very well, yet, since it is very hard to find a good bridge between real-life research and interesting science fiction that doesn't immediately get sidetracked to focus on the science fiction parts. However, in the instances in which it worked, this helpe... (read more)

I am looking for a website that presents bite-size psychological insights. Does anyone know such a thing?

I found the site http://www.psych2go.net/ in the past few days and I find the idea very appealing, since it is a very fast and efficient way to learn or refresh knowledge of psychological facts. Unfortunately, that website itself doesn't seem all that good since most of its feed is concerned with dating tips and other noise rather than actual psychological insights. Do you know something that is like it, but better and more serious?

The AI in that story actually seems to be surprisingly well done and does have an inherent goal to help humanity. It's primary goal is to 'satisfy human values through friendship and ponies'. That's almost perfect, since here 'satisfying human values' seems to be based on humanity's CEV.

It's just that the added 'through friendship and ponies' turns it from a nigh-perfect friendly AI into something really weird.

I agree with your overall point, though.

I would find it very interesting if the tournament had multiple rounds and the bots were able to adapt themselves based on previous performance and log files they generated at runtime. This way they could use information like 'most bots take longer to simulate than expected.' or 'there are fewer cannon-fodder bots than expected' and become better adapted in the next round. Such a setup would lessen the impact of the fact that some bots that are usually very good underperform here because of an unexpected population of competitors. This might be hard to implement and would probably scare away some participants, though.

I wouldn't call an AI like that friendly at all. It just puts people in utopias for external reasons, but it has no actual inherent goal to make people happy. None of these kinds of AIs are friendly, some are merely less dangerous than others.

I know this was just a harmless typo, and this is not intended as an attack, but I found the idea of a "casual" decision theory hilarious.

Then I noticed that that actually explains a great deal. Humans really do make decisions in a way that could be called casual, because we have limited time and resources and will therefore often just say 'meh, sounds about right' and go with it instead of calculating the optimal choice. So, in essence 'causal decision theory' + 'human heuristics and biases' = 'casual decision theory'

Yes, I was referring to LessWrong, not AI researchers in general.

No, it can't be done by brute-force alone, but faster hardware means faster feedback and that means more efficient research.

Also, once we have computers that are fast enough to just simulate a human brain, it becomes comparatively easy to hack an AI together by just simulating a human brain and seeing what happens when you change stuff. Besides the ethical concerns, this would also be insanely dangerous.

I would argue that these two goals are identical. Unless humanity dies out first, someone is eventually going to build an AGI. It is likely that this first AI, if it is friendly, will then prevent the emergence of other AGI's that are unfriendly.

Unless of course the plan is to delay the inevitable for as long as possible, but that seems very egoistic since faster computers make will make it easier to build an unfriendly AI in the future, while the difficulty of solving AGI friendliness will not be substantially reduced.

While I think this is a good idea in principle, most of these slogans don't seem very effective because they suffer from the illusion of transparency. Consider what they must look like to someone viewing this from the outside:

"AI must be friendly" just sounds weird to someone who isn't used to the lingo of calling AI 'friendly'. I can't think of an alternative slogan for this, but there must be a better way to phrase that.

"Ebola must die!" sounds great. It references a concrete risk that people understand and calls for its destruction. ... (read more)

I know, and that is part of what makes this so hard. Thankfully, I have several ways too cheat:

-I can take days thinking of the perfect path of action for what takes seconds in the story.

-The character is a humanoid avatar of a very smart and powerful entity. While it was created with much specialized knowledge, it is still human-like at its core.

But most importantly:

-It's a story about stories and there is an actual narrator-like entity changing the laws of nature. Sometimes, 'because this would make for a better story' is a perfectly valid criterion for choosing actions. The super-human characters are all aware of this and exploit it heavily.

I'm not sure I understand what you mean. Implement what functionality where? I don't think I'm going to start working for that company just because this feature is interesting :-) As for my own program, I changed it to use a health bar today, but that is of no use to anyone else, since the program is not designed to be easily usable by other people. I always find it terrible to consider that large companies have so many interdependencies that they take months to implement (and verify and test) what took an hour for my primitive program.

I heard of NaNoWriMo before. Unfortunately that would be too much for me to handle. I am not a professional writer. I am just doing this in my free time and I just don't have that kind of time, although I think this would definitely be worth checking out if it was during a holiday.

Yes, it's pretty similar. I think their idea of making the punishment affect a separate health bar rather than reducing the experience directly may actually be better. I should try that out some time. Unlike HabitRPG (I think?) my program is also a todo list, though. I use it for organizing my tasks and any task that I don't finish in time costs experience, just like failing a habit. This helps to prevent procrastination.

Thanks, these look really useful. I will definitely have a look at them.