The way it is now, when one lab has an insight, the insight will probably spread quickly to all the other labs. If we could somehow "drive capability development into secrecy," that would drastically slow down capability development.

Malice is a real emotion, and it is a bad sign (but not a particularly strong sign) if a person has never felt it.

Yes, letting malice have a large influence on your behavior is a severe character flaw, that is true, but that does not mean that never having felt malice or being incapable of acting out of malice is healthy.

Actually, it is probably rare for a person never to act out of malice: it is probably much more common for a person to just be unaware of his or her malicious motivations.

The healthy organization is to be tempted to act maliciously now and then, but to be good at perceiving when that is happening and to ignore or to choose to resist the temptation most of the time (out of a desire to be a good person).

I expect people to disagree with this answer.

their >90% doom disagrees with almost everyone else who thinks seriously about AGI risk.

The fact that your next sentence refers to Rohin Shah and Paul Christiano, but no one else, makes me worry that for you, only alignment researchers are serious thinkers about AGI risk. Please consider that anyone whose P(doom) is over 90% is extremely unlikely to become an alignment researcher (or to remain one if their P(doom) became high when they were an alignment researcher) because their model will tend predict that alignment research is futile or that it actually increases P(doom).

There is a comment here (which I probably cannot find again) by someone who was in AI research in the 1990s, then he realized that the AI project is actually quite dangerous, so he changed careers to something else. I worry that you are not counting people like him as people who have thought seriously about AGI risk.

I disagree. I think the fact that our reality branches a la Everett has no bearing on our probability of biogensis.

Consider a second biogenesis that happened recently enough and far away enough that light (i.e., information, causal influence) has not had enough time to travel from it to us. We know such regions of spacetime "recent enough and far away enough" exist and in principle could host life, but since we cannot observe a sign of life or a sign of lack of life from them, they are not relevant to our probability of biogenesis whereas by your logic, they are relevant.

new cities like Los Alamos or Hanover

You mean Hanford.

When you write,

a draft research paper or proposal that frames your ideas into a structured and usable format,

who is "you"?

I know you just said that you don't completely trust Huberman, but just today, Huberman published a 30-minute video titled "Master your sleep and be more alert when awake". I listened to it (twice) to refresh my memory and to see if his advice changed.

He mentions yellow-blue (YB) contrasts once (at https://www.youtube.com/watch?v=lIo9FcrljDk&t=502s) and at least thrice he mentions the desirability of exposure to outdoor light when the sun is at a low angle (close to the horizon). As anyone can see by looking around at dawn and again at mid-day, at dawn some parts of the sky will be yellowish (particularly, the parts of the sky near the sun) or even orange whereas other parts will range from pale blue to something like turquoise to deep blue whereas at mid-day the sun is white, the part of the sky near the sun is blue and the blue parts of the sky are essentially all the same shade or hue of blue.

He also says that outdoor light (directly from the sun or indirectly via atmospheric scattering) is the best kind of light for maintaining a healthy circadian rhythm, but that if getting outdoors early enough that the sun is still low in the sky is impractical, artificial light can be effective, particularly blue-heavy artificial light.

I've been help greatly over the last 2 years by a protocol in which I get outdoor light almost every morning when the YB contrasts are at its most extreme, namely between about 20 min before sunrise and about 10 min after sunrise on clear days and a little later on cloudy days. (The other element of my protocol that I know to be essential is strictly limit my exposure to light between 23:00 and 05:00.) I was motivated to comment on your post because it did not contain enough information to help someone sufficiently similar to me (the me of 2 years ago) to achieve the very welcome results I achieved: I'm pretty sure that even very bright artificial light from ordinary LED lights that most of us have in our home (even very many of them shining all at once) would not have helped me nearly as much.

Huberman is not so insistent on getting outside during this 30-minute interval of maximum YB contrast as my protocol is. In fact in today's video he says that he himself often gets outside only after the sun has been out for an hour or 2 and is consequently no longer particularly near the horizon.

Health-conscious people apply a (software-based) filter to their screens in the evening to reduce blue light emitted from the screen. On iOS this is called Night Shift. If your rendition of the effects of light on the circadian rhythm (CR) is complete, then they're doing everything they can do, but if YB contrasts have important effects on the CR, it might be useful in addition to eliminate YB contrasts on our digital devices (which Night Shift and its analogs on the other platforms do not eliminate). This can be done by turning everything shades of gray. (On iOS for example, this can be achieved in Settings > Accessibility > Display & Text Size > Color Filters > Grayscale and can be combined with or "overlaid on" Night Shift.) I and others do this (turn of a filter that makes everything "grayscale") routinely to make it more likely that we will get sleepy sufficiently early in the evening. Additional people report that they like to keep their screens grayscale, but do not cite the CR as the reason for their doing so.

Is a computer screen bright enough such that YB contrasts on the screen can activate the machinery in the retina that is activated by a sunrise? I'm not sure, but I choose to eliminate YB contrasts on my screens just in case it is.

Finally let me quote what I consider the main takeaway from the video Huberman posted today, which I expect we both agree with:

Get up each morning, try to get outside. I know that can be challenging for people, but anywhere from 2 to 10 min of sun exposure will work well for most people. If you can't do it every day or if you sleep through this period of early-day low-solar angle, don't worry about it. The systems in the body -- these hormone systems and neurotransmitter systems -- that make you awake at certain periods of the day and sleepy at other times are operating by averaging when you view the brightest light.

https://www.youtube.com/watch?v=lIo9FcrljDk&t=816s

This post paints a partially inaccurate picture. IMHO the following is more accurate.

Unless otherwise indicated, the following information comes from Andrew Huberman. Most comes from Huberman Lab Podcast #68. Huberman opines on a great many health topics. I want to stress that I don't consider Huberman a reliable authority in general, but I do consider him reliable on the circadian rhythm and on motivation and drive. (His research specialization for many years was the former and he for many years has successfully used various interventions to improve his own motivation and drive, which is very high.)

Bright light (especially bluish light) makes a person more alert. (Sufficiently acute exposure to cold, e.g., a plunge into 45-degree water, is an even stronger cause of alertness. Caffeine is another popular intervention for causing alertness.) After many hours of being alert and pursuing goals, a person will get tired, and this tiredness tends to help the person go to sleep. However, the SCN operates independently of exposure to bright light (and cold and caffeine) and independently of how many hours in a row the person has already been alert. A good illustration of that is what happens when a person pulls an all-nighter: at about 4:30 it becomes easier for most people pulling an all-nighter to stay awake even if the person is not being exposed to bright light and even if the person has already been awake for a very long time. Without any light as a stimulus, at around 04:30 the SCN decides to get the brain and the body ready for wakefulness and activity. So, let us inquire how the SCN would stay in sync with the sun in the ancestral environment before the availability of artificial lighting. How does the SCN know that dawn is coming soon?

The answer is that it is complicated, like most things in biology, but I think most neuroscientists agree that the stimulus that most potently entrains the SCN (i.e., that is most effective at ensuring the the SCN is in sync with the sun) is yellow-blue (YB) contrasts. Specifically, the SCN knows it is 4:30 and consequently time to start making the body alert because of the person's exposure to these "YB contrasts" on previous days. Exposure in the evening has an effect, but the strongest effect is exposure circa dawn.

When the sun is at a high angular elevation, it is white and the surrounding sky is dark blue (assuming a cloudless sky). When the sun is slightly above or below the horizon, the part of the sky near the sun is yellow or even orange or pink and with further (angular) distance from the sun, the sky gets steadily bluer. (Note that even 30 minutes before sunrise, the sky is already much brighter than your cell phone's screen or most indoor environments: there is an optical illusion whereby people underestimate the brightness of a light source when the source is spread over a large (angular) area and overestimate the brightness of "point sources" like light bulbs.)

The sensing of these YB contrasts is done by a system distinct from the usual visual system (i.e., the system that gives visual information that is immediately salient to the conscious mind) and in particular there are fewer "sensing pixels" and they are spread further apart than the "pixels" in the usual visual system. The final page of this next 5-page paper has a nice image of the author's estimate of a bunch of "sensing pixels" depicted as dotted circles laid over a photo of a typical sunrise:

https://pmc.ncbi.nlm.nih.gov/articles/PMC8407369/pdf/nihms-1719642.pdf

A light bulb named Tuo is recommended at least half-heartedly by Huberman for controlling the circadian rhythm. Huberman says IIRC it works by alternating between yellow light and blue light many times a second. Huberman explained IIRC that both "spatial" YB contrasts and "temporal" YB contrasts serve a signal that "dawn or dusk is happening". I definitely recall Huberman saying that outdoor light is preferred to this Tuo light bulb, and I understood him to mean that it is more likely to work because no one understands SCN entrainment well enough right now to design an artificial light source guaranteed to work.

The web site for this light bulb says,

Most light therapy products on the market today are based on blue light. This is 15-year-old science that has since been proven incorrect. New science based on laboratory research conducted by the University of Washington, one of the world's top vision and neuroscience centers, shows that blue light has little to no effect in shifting your circadian rhythm. High brightness levels of blue light can have some effect, but this falls short when compared to the power of TUO.

High lux products can work, but they require 10,000 lux of light at a distance of under 2 feet for the duration of treatment. This light level at this distance is uncomfortable for most people. Treatment also needs to happen first thing in the morning to be most effective. Who has time to sit within 2 feet of a bulb for up to a half hour when first waking up? High lux products offer dim settings to make use more comfortable and typically downplay the distance requirement. Unfortunately, at less than 10,000 lux and 2 feet of distance, high lux products have little to no impact.

Huberman recommends getting as much bright light early in the day as possible -- "preferably during the first hour after waking, but definitely during the first 3 hours". This encourages the body to produce cortisol and dopamine, which at this time of day is very good for you (and helps you be productive). But this bright light won't have much effect on keeping your circadian clock entrained with the schedule you want it be entrained with unless the light contains YB contrasts; i.e., getting bright light right after waking is good and getting sufficiently-bright light containing YB contrasts right after waking is also good, but they are good for different reasons (though the first kind of good contributes to a small extent to the second kind of good through the mechanism I described in my second paragraph).

Huberman is insistent that it is not enough to expose yourself to light after your normal wake-up time: you also have to avoid light when you are normally asleep. Suppose your normal wake-up time is 06:00. To maintain a strong circadian rhythm and to get to sleep at a regular time each night (which is good for you and which most people should strive to do) it is essential to avoid exposure to light during the 6 hours between 23:00 and 05:00. Whereas dim light has little positive effect after wake-up time, even quite dim light or light of brief duration between 23:00 and 05:00 tends to have pronounced negative effects.

Light during these 6 hours not only confuses the circadian clock (which is bad and makes it hard to get to sleep at a healthy hour) but it also decreases the amount of motivation and drive available the next morning (by sending signals to a brain region called the habenula). I personally have noticed a strong increase in my level of motivation and drive on most mornings after I instituted the habits described in this comment. (And I more reliably start my sleep at what I consider a healthy hour, but that was less critical in my case because insomnia was never a huge problem of mine.)

Huberman says that getting outside at dawn works to keep the SCN in sync with the sun even on completely overcast days, but it requires longer duration of exposure: 20 minutes instead of 5 minutes IIRC. He says that there are YB contrasts in the overcast sky when the sun is near the horizon that are absent when the angle of the sun is high.

To this point in this comment I merely repeated information I learned from Huberman (and maybe a bit from Wikipedia or such -- it is hard to remember) although I hasten to add that this information certainly jibes with my own experience of going outside at dawn almost every day starting about 2 years ago. Allow me to add one thing of my own invention, namely, what to call this 6-hour interval every night when it is a bad idea to let your eyes be exposed to light: I humbly suggest "curfew". The original meaning of "curfew" was a time every night during which it was illegal in medieval London to have a fire going even in your own fireplace in your own home. (I.e., it was a measure to prevent urban fires.)

“Game theoretic strengthen-the-tribe perspective” is a completely unpersuasive argument to me. The psychological unity of humankind OTOH is persuasive when combined with the observation that this unitary psychology changes slowly enough that the human mind’s robust capability to predict the behavior of conspecifics (and manage the risks posed by them) can keep up.

Although I agree with another comment that Wolfram has not "done the reading" on AI extinction risk, my being able to watch his face while he confronts some of the considerations and arguments for the first time made it easier, not harder, for me to predict where his stance on the AI project will end up 18 months from now. It is hard for me to learn anything about anyone by watching them express a series of cached thoughts.

Near the end of the interview, Wolfram say that he cannot do much processing of what was discussed "in real time", which strongly suggests to me that he expects to process it slowly over the next days and weeks. I.e., he is now trying to reassure himself that the AI project won't kill his four children or any grandchildren he has or will have. Because Wolfram is much better AFAICT at this kind of slow "strategic rational" deliberation than most people at his level of life accomplishment, there is a good chance he will fail to find his slow deliberations reassuring, in which case he probably then declares himself an AI doomer. Specifically, my probability is .2 that 18 months from now, Wolfram will have come out publicly against allowing ambitious frontier AI research to continue. P = .2 is much much higher than my P for the average 65-year-old of his intellectual stature who is not specialized in AI. My P is much higher mostly because I watched this interview; i.e., I was impressed by Wolfram's performance in this interview despite his spending the majority of his time on rabbit holes than I could quickly tell had no possible relevance to AI extinction risk.

My probability that he will become more optimistic about the AI project over the next 18 months is .06: mostly likely, he goes silent on the issue or continues to take an inquisitive non-committal stance in his public discussions of it.

If Wolfram had a history of taking responsibility for his community, e.g., campaigning against drunk driving or running for any elected office, my P of his declaring himself an AI doomer (i.e., becoming someone trying to stop AI) would go up to .5. (He might in fact have done something to voluntarily take responsibility for his community, but if so, I haven't been able to learn about it.) If Wolfram were somehow forced to take sides, and had plenty of time to deliberate calmly on the choice after the application of the pressure to choose sides, he would with p = .88 take the side of the AI doomers.