Open Thread Fall 2024

habryka

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.

If it’s worth saying, but not worth its own post, here's a place to put it.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

The Open Thread tag is here. The Open Thread sequence is here.

I found the site a few months ago due to a link from an AI themed forum. I read the sequences and developed the belief that this was a place for people who think in ways similar to me. I work as a nuclear engineer. When I entered the workforce, I was surprised to find that there weren’t people as dispositioned toward logic as I was. I thought perhaps there wasn’t really a community of similar people and I had largely stopped looking.

This seems like a good place for me to learn, for the time being. Whether or not this is a place for me to develop community remains to be seen. The format seems to promote people presenting well-formed ideas. This seems valuable, but I am also interested in finding a space to explore ideas which are not well-formed. It isn’t clear to me that this is intended to be such a space. This may simply be due to my ignorance of the mechanics around here. That said, this thread seems to be inviting poorly formed ideas and I aim to oblige.

There seem to be some writings around here which speak of instrumental rationality, or “Rationality Is Systematized Winning”. However, this seems to beg the question: “At what scale?” My (perhaps naive) impression is that if you execute instrumental rationality with an objective function at the personal scale it might yield the decision that one should go work in finance and accrue a pile of utility. But if you apply instrumental rationality to an objective function at the societal scale it might yield the decision to give all your spare resources to the most effective organizations you can find. It seems to me that the focus on rationality is important but doesn’t resolve the broader question of “In service of what?” which actually seems to be an important selector of who participates in this community. I don’t see much value in pursuing Machiavellian rationality and my impression is that most here don’t either. I am interested in finding additional work that explores the implications of global scale objective functions.

On a related topic, I am looking to explore how to determine the right scale of the objective function for revenge (or social correction if you prefer a smaller scope). My intuition is that revenge was developed as a mechanism to perform tribal level optimizations. In a situation where there has been a social transgression, and redressing that transgression would be personally costly but societally beneficial, what is the correct balance between personal interest and societal interest?

My current estimate of P(doom) in the next 15 years is 5%. That is, high enough to be concerned , but not high enough to cash out my retirement. I am curious about anyone harboring a P(doom) > 50%. This would seem to be high enough to support drastic actions. What work has been done to develop rational approaches to such a high P(doom)?

This idea is quite poorly formed, but I am interested in exploring how to promote encapsulation, specialization, and reuse of components via the cost function in an artificial neural network. This comes out of the intuition that actions (things described by verbs, or transforms) may be a primitive in human mental architecture and are one of the mechanisms by which analogical connections are searched. I am interested in seeing if continuous mechanisms could be defined to promote the development of a collection of transforms which could be applied usefully across multiple different domains. Relatedly, I am also interested in what an architecture/cost function would need to look like to promote retaining multiple representations of a concept with differing levels of specificity/complexity.

Use a smart but not self-improving AI agent to antagonize the world with the goal of making advanced societies believe that AGI is a bad idea and precipitating effective government actions. You could call this the Ozymandias approach.

ChaosGPT already exists. It's incompetent to the point of being comical at the moment, but maybe more powerful analogues will appear and wreak havoc. Considering the current prevalence of malware, it might be more surprising if something like this didn't happen.

We've already seen developments that could have been considere... (read more)

3gilch2y

We have already identified some key resources involved in AI development that could be restricted. The economic bottlenecks are mainly around high energy requirements and chip manufacturing. Energy is probably too connected to the rest of the economy to be a good regulatory lever, but the U.S. power grid can't currently handle the scale of the data centers the AI labs want for model training. That might buy us a little time. Big tech is already talking about buying small modular nuclear reactors to power the next generation of data centers. Those probably won't be ready until the early 2030s. Unfortunately, that also creates pressures to move training to China or the Middle East where energy is cheaper, but where governments are less concerned about human rights. A recent hurricane flooding high-purity quartz mines made headlines because chip producers require it for the crucibles used in making silicon wafers. Lower purity means accidental doping of the silicon crystal, which means lower chip yields per wafer, at best. Those mines aren't the only source, but they seem to be the best one. There might also be ways to utilize lower-purity materials, but that might take time to develop and would require a lot more energy, which is already a bottleneck. The very cutting-edge chips required for AI training runs require some delicate and expensive extreme-ultraviolet lithography machines to manufacture. They literally have to plasmify tin droplets with a pulsed laser to reach those frequencies. ASML Holdings is currently the only company that sells these systems, and machines that advanced have their own supply chains. They have very few customers, and (last I checked) only TSMC was really using them successfully at scale. There are a lot of potential policy levers in this space, at least for now.

4gilch2y

The instrumental convergence of goals implies that a powerful AI would almost certainly act to prevent any rivals from emerging, whether aligned or not. In the intelligence explosion scenario, progress would be rapid enough that the first mover achieves a decisive strategic advantage over the entire world. If we find an alignment solution robust enough to survive the intelligence explosion, it will set up guardrails to prevent most catastrophes, including the emergence of unaligned AGIs. [...] Alignment and capabilities don't necessarily correlate, and that accounts for lot of why my p(doom) is so high. But more aligned agents are, in principle, more useful, so rational organizations should be motivated to pursue aligned AGI, not just AGI. Unfortunately, alignment research seems barely tractable, capabilities can be brute-forced (and look valuable in the short term) and corporate incentive structures being what they are, in practice, what we're seeing is a reckless amount of risk taking. Regulation could alter the incentives to balance the externality with appropriate costs.

44

Open Thread Fall 2024

44

44

44

Open Thread Fall 2024

44

44