Wiki Contributions

Comments

Your post reads a bit strangely. 

At first, I thought you were arguing that AGI might be used by some extremists to wipe out most of humanity for some evil and/or stupid reason.  Which does seem like a real risk.  

Then you went on to point out that someone who thought that was likely might wipe out most of humanity (not including themselves) as a simple survival strategy, since otherwise someone else will wipe them out (along with most other people). As you note, this requires a high level of unconcern for normal moral considerations, which one would think very few people would countenance.

Now comes the strange part... You argue that actually maybe many people would be willing to wipe out most of humanity to save themselves, because...  wiping out most of humanity sounds like a pretty good idea!

I'm glad that in the end you seem to still oppose wiping out most of humanity, but I think you have some factual misconceptions about this, and correcting them is a necessary first step to thinking of how to address the problem.

Concerning climate change, you write: "In the absence of any significant technological developments, sober current trajectory predictions seem to me to range from 'human extinction' to 'catastrophic, but survivable'".

No. Those are not "sober" predictions. They are alarmist claptrap with no scientific basis. You have been lied to. Without getting into details, you might want to contemplate that global temperatures were probably higher than today during the "Holocene Climatic Optimum" around 8000 years ago.  That was the time when civilization developed.  And temperatures were significantly higher in the previous interglacial, around 120,000 years ago.  And the reference point for supposedly-disastrous global warming to come is "pre-industrial" time, which was in the "little ice age", when low temperatures were causing significant hardship. Now, I know that the standard alarmist response is that it's the rate of change that matters.  But things changed pretty quickly at the end of the last ice age, so this is hardly unprecedented. And you shouldn't believe the claims made about rates of change in any case - actual science on this question has stagnated for decades, with remarkably little progress being made on reducing the large uncertainty about how much warming CO2 actually causes.

Next, you say that the modern economy is relatively humane "under conditions of growth, which, under current conditions, depends on a growing population and rising consumption. Under stagnant or deflationary conditions it can be expected to become more cutthroat, violent, undemocratic and unjust."

Certainly, history teaches that a social turn towards violence is quite possible. We haven't transcended human nature.  But the idea that continual growth is needed to keep the economy from deteriorating just has no basis in fact.  Capitalist economies can operate perfectly fine without growth.  Of course, there's no guarantee that the economy will be allowed to operate fine.  There have been many disastrous economic policies in the past.  Again, human nature is still with us, and is complicated. Nobody knows whether social degeneration into poverty and tyranny is more likely with growth or without growth.

Finally, the idea that a world with a small population will be some sort of utopia is also quite disconnected from reality.  That wasn't the way things were historically. And even if it was, it woudn't be stable, since population will grow if there's plenty of food, no disease, no violence, etc. 

So, I think your first step should be to realize that wiping out most of humanity would not be a good thing. At all. That should make it a lot easier to convince other people not to do it.

It probably doesn't matter, but I wonder why you used the name "Sam" and then referred to this person as "she".  The name "Sam" is much more common for men than for women. So this kicks the text a bit "out of distribution", which might affect things. In the worst case, the model might think that "Sam" and "she" refer to different people.

There are in fact many universities that have both "research faculty" and "teaching faculty".  Being research faculty has higher prestige, but nowadays it can be the case that teaching faculty have almost the same job security as research faculty.  (This is for permanent teaching faculty, sessional instructors have very low job security.)

In my experience, the teaching faculty often do have a greater enthusiasm for teaching than most research faculty, and also often get better student evaluations.  I think it's generally a good idea to have such teaching faculty.

However, my experience has been that there are some attitudinal differences that indicate that letting the teaching faculty have full control of the teaching aspect of the university's mission isn't a good idea.

One such is a tendency for teaching faculty to start to see the smooth running of the undergraduate program as an end in itself.  Research faculty are more likely to have an ideological commitment to the advancement of knowledge, even if promoting that is not as convenient.

A couple anecdotes (from my being research faculty at a highly-rated university):

At one point, there was a surge in enrollment in CS. Students enrolled in CS programs found it hard to take all the courses they needed, since they were full.  This led some teaching faculty to propose that CS courses (after first year) no longer be open to students in any other department, seeing as such students don't need CS courses to fulfill their degree requirements. Seems logical: students need to smoothly check off degree requirements and graduate. The little matter that knowledge of CS is crucial to cutting-edge research in many important fields like biology and physics seemed less important...

Another time, I somewhat unusually taught an undergrad course a bit outside my area, which I didn't teach again the next year.  I put all the assignments I gave out, with solutions, on my web page.  The teaching faculty instructor the next year asked me to take this down, worrying that students might find answers to future assigned questions on my web page. I pointed out that these were all my own original questions, not from the textbook, and asked whether he also wanted the library to remove from circulation all the books on this topic... 

Also, some textbooks written by teaching faculty seem more oriented towards moving students through standard material than teaching them what is actually important. 

Nevertheless, it is true that many research faculty are not very good at teaching, and often not much interested either.  A comment I once got on a course evaluation was "there's nothing stupid about this course".  I wonder what other experiences this student had had that made that notable!

These ideas weren't unfamiliar to Hinton.  For example, see the following paper on "Holographic Reduced Representations" by a PhD student of his from 1991: https://www.ijcai.org/Proceedings/91-1/Papers/006.pdf

The logic seems to be:

  1. If we do a 1750 year simulation assuming yearly fresh water additions 80 times the current greenland ice melt rate, we see AMOC collapse.
  2. Before this simulated collapse, the value of something that we think could be an indicator changes.
  3. That indicator has already changed.
  4. So collapse of the AMOC is imminent.

Regarding (1), I think one can assume that if there was any way of getting their simulation engine to produce an AMOC collapse in less than 1750 years, they would have showed that.  So, to produce any sort of alarming result, they have to admit that their simulation is flawed, so they can say that collapse might in reality occur much sooner.  But then, if the simulation is so flawed, why would one think that the simulation's indicator has any meaning? 

They do claim that the indicator isn't affected by the simulation's flaws, but without having detailed knowledge to assess this myself, I don't see any strong reason to believe them.  It seems very much like a paper that sets out to show what they want to show.

From the paper:

Under increasing freshwater forcing, we find a gradual decrease (Fig. 1A) in the AMOC strength (see Materials and Methods). Natural variability dominates the AMOC strength in the first 400 years; however, after model year 800, a clear negative trend appears because of the increasing freshwater forcing. Then, after 1750 years of model integration, we find an abrupt AMOC collapse

Given that the current inter-glacial period would be expect to last only on the order of some thousands of years more, this collapse in 1750 years seems a bit academic. 

I don't get it.

Apparently, the idea is that this sort of game tells us something useful about AI safety.

But I don't get it.

You obviously knew that you were not unleashing a probably-malign superintelligence on the the world by letting Ra out.  So how does your letting Ra out in this game say anything about how you would behave if you did think that (at least initially)?

So I don't get it.

And if this does say something useful about AI safety, why is it against the rules to tell us how Ra won?

I don't get it.

Interesting.  I hadn't heard of the Child Born on Tuesday Problem.  I think it's actually quite relevant to Sleeping Beauty, but I won't go into that here...

Both problems (your 1 and 2) aren't well-defined, however. The problem is that in real life we do not magically acquire knowledge that the world is in some subset of states, with the single exception of the state of our direct sense perceptions. One could decide to assume a uniform distribution over possible ways in which the information we are supposedly given actually arrives by way of sense perceptions, but uniform distributions are rather arbitrary (and will often depend on arbitrary aspects of how the problem is formulated).

Here's a boys/girls puzzle I came up with to illustrate the issue:

 A couple you've just met invite you over to dinner, saying "come by around 5pm, and we can talk for a while before our three kids come home from school at 6pm".

You arrive at the appointed time, and are invited into the house. Walking down the hall, your host points to three closed doors and says, "those are the kids' bedrooms".  You stumble a bit when passing one of these doors, and accidentally push the door open.  There you see a dresser with a jewelry box, and a bed on which a dress has been laid out.  "Ah", you think to yourself, "I see that at least one of their three kids is a girl".

Your hosts sit you down in the kitchen, and leave you there while they go off to get goodies from the stores in the basement.  While they're away, you notice a letter from the principal of the local school tacked up on the refrigerator.  "Dear Parent", it begins, "Each year at this time, I write to all parents, such as yourself, who have a boy or boys in the school, asking you to volunteer your time to help the boys' hockey team..."  "Umm", you think, "I see that they have at least one boy as well".

That, of course, leaves only two possibilities:  Either they have two boys and one girl, or two girls and one boy.  What are the probabilities of these two possibilities?

 The symmetrical summaries of what is learned are intentionally misleading (it's supposed to be a puzzle, after all).  The way in which you learned they have at least one girl is not the same as the way you learned that they have at least one boy. And that matters.

You may think the difference between "the card is an Ace" and "JeffJo says the card is an Ace" is just a quibble.  But this is actually a very common source of error.  

Consider the infamous "Linda" problem, in which researchers claim that most people are irrational because they think "Linda is a bank teller" is less likely than "Linda is a bank teller and active in the feminist movement".  When you think most people are this blatantly wrong, you maybe need to consider that you might be the one who's confused...

Actually, there is no answer to the problem as stated. The reason is that the evidence I (who drew the card) have is not "the card is an Ace", but rather "JeffJo said the card is an Ace". Even if I believe that JeffJo never lies, this is not enough to produce a probability for the card being the Ace of Spades. I would need to also consider my prior probability that JeffJo would say this conditional on it being the Ace of Space, the Ace of Hearts, the Ace of Diamonds, or the Ace of Clubs. Perhaps I believe the JeffJo would never say the card is an Ace if it is a Space. In that case, the right answer is 0.

However, I agree that a "reward structure" is not required, unless possible rewards are somehow related to my beliefs about what JeffJo might do.

For example, I can assess my probability that the store down the street has ice cream sundaes for sale when I want one, and decide that the probability is 3/4. If I then change my mind and decide that I don't want an ice cream sundae after all, that should not change my probability that one is available.

Load More