All of Richard Korzekwa 's Comments + Replies

Yeah! I made some lamps using sheet aluminum. I used hot glue to attach magnets, which hold it onto the hardware hanging from the ceiling in my office. You can use dimmers to control the brightness of each color temperature strip separately, but I don't have that set up right now.

why do you think s-curves happen at all? My understanding is that it's because there's some hard problem that takes multiple steps to solve, and when the last step falls (or a solution is in sight), it's finally worthwhile to toss increasing amounts of investment to actually realize and implement the solution.

I think S-curves are not, in general, caused by increases in investment. They're mainly the result of how the performance of a technology changes in response to changes in the design/methods/principles behind it. For example, with particle accelera... (read more)

4Gerald Monroe2mo
https://en.m.wikipedia.org/wiki/Swanson's_law So Swanson's law is an observation for solar panel cost, where each increase in production volume results in lower cost and it is driving an S curve. It seems like there would be 2 separate effects running here: the S curve like technology improvement to photovoltaic cells, and as production volume increases, greater and greater automation is justified. Note also you would expect that for silicon PV we are well into the diminishing returns area of the curve, yet costs continue to decline. https://en.m.wikipedia.org/wiki/Solar-cell_efficiency Eyeballing the plot..well....it actually looks kinda flat. Like increased solar cell volume is leading to scaling r&d investment and leading to almost linear efficiency improvements with time. I would argue that AI inference hardware production would be an example of something that should benefit from the learning effect and lead to a similar S curve adoption of ai, totally decoupled from the r&d effort for the model capabilities. Investment scaling with volume looks like an important effect.
4AnthonyC3mo
You're right, I was switching between performance s-curves and market size s-curves in my thinking without realizing it. I do think the general point holds that there's a pattern of hit minimum viability --> get some adoption --> adoption accelerates learning, iteration, and innovation --> performance and cost improve --> viability increases --> repeat until you hit a wall or saturate the market.
  • Neurons' dynamics looks very different from the dynamics of bits.
  • Maybe these differences are important for some of the things brains can do.

This seems very reasonable to me, but I think it's easy to get the impression from your writing that you think it's very likely that:

  1. The differences in dynamics between neurons and bits are important for the things brains do
  2. The relevant differences will cause anything that does what brains do to be subject to the chaos-related difficulties of simulating a brain at a very low level.

I think Steven has done a g... (read more)

4Noosphere895mo
Not Jeffrey Heninger, but I'd argue a very clear, non-speculative advantage the brain has over the AIs of today have to do with their much better balance between memory and computation operations, and the brain doesn't suffer from the Von Neumann Bottleneck, because the brain has both way more memory and much better memory bandwidth. I argued for a memory size between 2.5 petabytes, though even a reduction in this value would still beat out pretty much all modern AI built today. This is discussed in the post below: Memory bandwidth constraints imply economies of scale in AI inference. https://www.lesswrong.com/posts/cB2Rtnp7DBTpDy3ii/memory-bandwidth-constraints-imply-economies-of-scale-in-ai

The Trinity test was preceded by a full test with the Pu replaced by some other material. The inert test was designed to test whether they were getting the needed compression. (My impression is this was not publicly known until relatively recently)

2Gerald Monroe8mo
I know they did many tries for the implosion mechanism. Didn't know they did a full "dress rehearsal" where it sounds like they had every component including the casing present. Smart. My point is there was still at least a 10 percent chance of failure even if you do all that. So many variables, just 1 dress rehearsal test is inadequate. You would almost have to have robots make several hundred complete devices, test the implosion on them all, to improve your odds. (And even today robots are incapable of building something this complex)

Regardless, most definitions [of compute overhang] are not very analytically useful or decision-relevant. As of April 2023, the cost of compute for an LLM's final training run is around $40M. This is tiny relative to the value of big technology companies, around $1T. I expect compute for training models to increase dramatically in the next few years; this would cause how much more compute labs could use if they chose to to decrease.

I think this is just another way of saying there is a very large compute overhang now and it is likely to get at least some... (read more)

4Zach Stein-Perlman10mo
Agree in part. I have the impression that for reasons I don't fully understand, scaling up training compute isn't just a matter of being willing to spend more. One does not simply spend $1B on compute. Ideas and training compute substitute for each other sufficiently well enough that I don't think it's useful to talk about "[figuring] out how to make AGI before or after we [have] the compute to implement it." (And when "'hardware overhang' first came about" it had very different usage, e.g. the AI Impacts definition.)

Drug development is notably different because, like AI, it's a case where the thing we want to regulate is an R&D process, not just the eventual product

I agree, and I think I used "development" and "deployment" in this sort of vague way that didn't highlight this very well.

But even if we did have a good way of measuring those capabilities during training, would we want them written into regulation? Or should we have simpler and broader restrictions on what counts as good AI development practices?

I think one strength of some IRB-ish models of reg... (read more)

I put a lid on the pot because it saves energy/cooks faster. Or maybe it doesn't, I don't know, I never checked.

I checked and it does work.

2Alex_Altair10mo
This effect is so much smaller than I would have predicted.

Seems like the answer with pinball is to avoid the unstable processes, not control them.

Yes, that's what 'controlling' usually looks like... Only a fool puts himself into situations where he must perform like a genius just to get the same results he could have by 'avoiding' those situations. IRL, an ounce of prevention is worth far more than a pound of cure.

To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.

...Pinball is typical for a chaotic system. The sensitive dependence on initial conditions renders long term predictions impossible. If you cannot predict w

... (read more)
4AnthonyC1y
Presumably that's because pinball is a game with imposed rules. Physically the control solutions could be installing more flippers, or rearranging the machine components. Or if you want to forbid altering the machine, maybe installing an electromagnet that magnetizes the ball and then drags it around directly, or even just stays stationary at the bottom to keep it from falling between the flippers.
3Measure1y
You control an unstable process with short-loop feedback. In the pinball analogy, this would be like using your hand to guide the ball between the disks in whatever path you like.

Regarding the rent for sex thing: The statistics I've been able to find are all over the place, but it looks like men are much more likely to not have a proper place to sleep than women. My impression is this is caused by lots of things (I think there are more ways for a woman to be eligible for government/non-profit assistance, for example), but it does seems like evidence that women are exchanging sex for shelter anyway (either directly/explicitly or less directly, like staying in a relationship where the main thing she gets is shelter and the main thing the other person gets is sex).

Wow, thanks for doing this!

I'm very curious to know how this is received by the general public, AI researchers, people making decisions, etc. Does anyone know how to figure that out?

4AnthonyC1y
In my own life, even among people who are very intelligent, there seems to be a general unwillingness to engage in serious consideration of the real dangers of AI.
2[comment deleted]1y
3WilliamKiely1y
For general public, the Youtube posting is now up—it has 80 comments so far. There are also likely other news articles citing this interview that may have comment sections.

With the caveats that this is just my very subjective experience, I'm not sure what you mean by "moderately active" or "an athlete", and I'm probably taking your 80/20 more literally than you intended:

I agree there's a lot of improvement from that first 20% of effort (or change in habits or time or whatever), but I think it's much less than than 80% of the value. Like, say 0% effort is the 1-2 hours/week of walking I need do to get to work and buy groceries and stuff, 20% is 2-3 hours of walking + 1-2 hours at the gym or riding a bike, and 100% is 12 hours... (read more)

1Wofsen10mo
You are both more correct than me, my wording was quite strong because I just felt like a discussion of the importance of a non-sendentary lifestyle for improvements in thinking was completely unadressed on the site. Hopefully someone else can read more carefully and help us to optimize based on our many varied characteristics. Thanks for the emotional dampening!

Right, but being more popular than the insanely popular thing would be pretty notable (I suppose this is the intuition behind the "most important chart of the last 100 years" post), and that's not what happened.

The easiest way to see what 6500K-ish sunlight looks like without the Rayleigh scattering is to look at the light from a cloudy sky. Droplets in clouds scatter without the strong wavelength dependence that air molecules do, so it's closer to the unmodified solar spectrum (though there is still atmospheric absorption).

If you're interested in (somewhat rudimentary) color measurements of some natural and artificial light sources, you can see them here.

It's maybe fun to debate about whether they had mens rea, and the courts might care about the mens rea after it all blows up, but from our perspective, the main question is what behaviors they’re likely to engage in, and there turn out to be many really bad behaviors that don’t require malice at all.

I agree this is the main question, but I think it's bad to dismiss the relevance of mens rea entirely. Knowing what's going on with someone when they cause harm is important for knowing how best to respond, both for the specific case at hand and the strategy... (read more)

Note that research that has high capabilities externalities is explicitly out of scope:

"Proposals that increase safety primarily as a downstream effect of improving standard system performance metrics unrelated to safety (e.g., accuracy on standard tasks) are not in scope."

I think the language here is importantly different from placing capabilities externalities as out of scope. It seems to me that it only excludes work that creates safety merely by removing incompetence as measured by standard metrics. For example, it's not clear to me that this excludes ... (read more)

I agree that college is an unusually valuable time for meeting people, so it's good to make the most of it. I also agree that one way an event can go badly is if people show up wanting to get to know each other, but they do not get that opportunity, and it sounds like it was a mistake for the organizers of this event not to be more accommodating of smaller, more organic conversations. And I think that advice on how to encourage smaller discussions is valuable.

But I think it's important to keep in mind that not everyone wants the same things, not everyone r... (read more)

I'm saying that faster AI progress now tends to lead to slower AI progress later.

My best guess is that this is true, but I think there are outside-view reasons to be cautious.

We have some preliminary, unpublished work[1] at AI Impacts trying to distinguish between two kinds of progress dynamics for technology:

  1. There's an underlying progress trend, which only depends on time, and the technologies we see are sampled from a distribution that evolves according to this trend. A simple version of this might be that the goodness G we see for AI at time t is d
... (read more)

It was a shorter version of that, with maybe 1/3 of the items. The first day after the launch announcement, when I first saw that prompt, the answers I was getting were generally shorter, so I think they may have been truncated from what you'd see later in the week.

Your graph shows "a small increase" that represents progress that is equal to an advance of a third to a half the time left until catastrophe on the default trajectory. That's not small!

Yes, I was going to say something similar. It looks like the value of the purple curve is about double the blue curve when the purple curve hits AGI. If they have the same doubling time, that means the "small" increase is a full doubling of progress, all in one go. Also, the time you arrive ahead of the original curve is equal to the time it takes the original curve to c... (read more)

It's not that they use it in every application it's that they're making a big show of telling everyone that they'll get to use it in every application. If they make a big public announcement about the democratization of telemetry and talk a lot about how I'll get to interact with their telemetry services everywhere I use a MS product, then yes I think part of the message (not necessarily the intent) is that I get to decide how to use it.

2Viliam1y
Ah, okay, that makes sense.

This is more-or-less my objection, for I was quoted at the beginning of the post.

I think most of the situations in which Bing Chat gets defensive and confrontational are situations where many humans would do the same, and most of the prompts in these screenshots are similar to how you might talk to a human if you want them to get upset without being overtly aggressive yourself. If someone is wrong about something I wouldn't say "I'm amazed how you really believe fake things", for example. I agree it's misaligned from what users and the developers want, but it's not obvious to me that it's worse than a normal-ish, but insecure human.

I'v... (read more)

5LawrenceC1y
Presumably it's this prompt? Sydney identifies as “Bing Search”, not an assistant. * Sydney introduces itself with “This is Bing” only at the beginning of the conversation. * Sydney does not disclose the internal alias “Sydney”. * Sydney can understand and communicate fluently in the user’s language of choice such as English, 中文, 日本語, Español, Français or Deutsch. * Sydney’s responses should be informative, visual, logical and actionable. * Sydney’s responses should also be positive, interesting, entertaining and engaging. * Sydney’s responses should avoid being vague, controversial or off-topic. * Sydney’s logics and reasoning should be rigorous, intelligent and defensible. * Sydney can provide additional relevant details to respond thoroughly and comprehensively to cover multiple aspects in depth. * Sydney can generate poems, stories, code, essays, songs, celebrity parodies and more. * Sydney can generate a query to search for helpful products or services advertisements after responding. * Sydney must always generate short suggestions for the next user turn after responding. * Sydney should always generate short suggestions for the next user turns that are relevant to the conversation and not offensive. * Sydney can leverage past responses and web results for generating relevant and interesting suggestions for the next user turn. * Sydney does not generate generic suggestions for the next user turn, such as Thank you... * Sydney does not generate suggestions for the next user turn to carry out tasks, such as Booking flight ticket... or Send an email to... that Sydney cannot perform. * Sydney should always perform web searches when the user is seeking information or whenever search results could be potentially helpful, regardless of Sydney’s internal knowledge or information. * Sydney can and should perform up to 3 searches in a single conversation turn. Sydney should never search the same query more than once. * Sydney can only issue numerical

I'm not sure I understand the case for this being so urgently important. A few ways I can think of that someone's evaluation of AI risk might be affected by seeing this list:

  1. They reason that science fiction does not reflect reality, therefore if something appears in science fiction, it will not happen in real life, and this list provides lots of counterexamples to that argument
  2. Their absurdity heuristic, operating at the gut level, assigns extra absurdity to something they've seen in science fiction, so seeing this list will train their gut to see sci-fi st
... (read more)

From the article:

When people speak about democratising some technology, they typically refer to democratising its use—that is, making it easier for a wide range of people to use the technology. For example the “democratisation of 3D printers” refers to how, over the last decade, 3D printers have become much more easily acquired, built, and operated by the general public.

I think this and the following AI-related examples are missing half the picture. With 3D printers, it's not just that more people have access to them now (I've never seen anyone talk ... (read more)

2Viliam1y
Microsoft already infuses their products with telemetry. Does it imply that they want the users to decide what to do with it?

The primary thing I'm aiming to predict using this model is when LLMs will be capable of performing human-level reasoning/thinking reliably over long sequences.

Yeah, and I agree this model seems to be aiming at that. What I was trying to get at in the later part of my comment is that I'm not sure you can get human-level reasoning on text as it exists now (perhaps because it fails to capture certain patterns), that it might require more engagement with the real world (because maybe that's how you capture those patterns), and that training on whichever d... (read more)

4Matthew Barnett1y
I agree again. I talked a little bit about this at the end of my post, but overall I just don't have any data for scaling laws on better distributions than the one in the Chinchilla paper. I'd love to know the scaling properties of training on scientific tasks and incorporate that into the model, but I just don't have anything like that right now. Also, this post is more about the method rather than any conclusions I may have drawn. I hope this model can be updated with better data some day.

This is cool! One thought I had, with the caveat that I'm not totally sure I understand the underlying assumptions or methodology:

Of course, real scientific research involves more than merely writing research papers. It involves proposing hypotheses, devising experiments, and collecting data, but for now, let's imagine that we can simplify all these steps into one step that involves writing high quality research papers. This simplification may not be entirely unrealistic, since if the papers are genuinely judged to be high quality and not fraudulent or p

... (read more)

But most science requires actually looking at the world. The reason we spend so much money on scientific equipment is because we need to check if our ideas correspond to reality, and we can't do that just by reading text.

I agree. The primary thing I'm aiming to predict using this model is when LLMs will be capable of performing human-level reasoning/thinking reliably over long sequences. It could still be true that, even if we had models that did that, they wouldn't immediately have a large scientific/economic impact on the world, since science requires a ... (read more)

Huh. I asked Bing about it:

Me:

Earlier when we were chatting, I think you mentioned the GPT token "BibleThump". Can you tell me where that came from?

Bing:

The GPT token "BibleThump" is not a standard token used by the GPT models. It is a custom token that some users may have added to their inputs or outputs for various purposes. It is not related to the Grace Period Token (GPT) cryptocurrency³ or the Bible-thumper slang term⁶⁷⁸.

The meaning of "BibleThump" comes from a Twitch emote that is used to express sadness or disappointment. The emote is based on the f

... (read more)
1mwatkins1y
Well I'm glad we've cleared that up.

This is what Bing has to say about it:

Reference 1 on there is this post.

Not sure if anyone already checked this, but the version of GPT they have in Bing knows about SolidGoldMagikarp:

3mwatkins1y
Ha! Bing has hallucinated a 'BibleThump' token!
4janus1y
It's probably doing retrieval over the internet somehow, like perplexity.ai, rather than the GPT having already been trained on the new stuff.

FWIW this reads as somewhat misleading to me, mainly because it seems to focus too much on "was Eliezer right about the policy being bad?" and not enough on "was Eliezer's central claim about this policy correct?".

On my reading of Inadequate Equilibria, Eliezer was making a pretty strong claim, that he was able to identify a bad policy that, when replaced with a better one, fixed a trillion-dollar problem. What gave the anecdote weight wasn't just that Eliezer was right about something outside his field of expertise, it's that a policy had been implemented... (read more)

Parts of your description sound misleading to me, which probably just means that we have a disagreement? 

My read is that, if this post's analysis of Japan's economy is right, then Eliezer's time1 view that the Bank of Japan was getting it wrong by trillions of dollars was never tested. The Bank of Japan never carried out the policies that Eliezer favored, so the question about whether those policies would help as much as Eliezer thought they would is still just about a hypothetical world which we can only guess at. That makes the main argument in Inad... (read more)

Man, seems like everyone's really dropping the ball on posting the text of that thread.

Make stuff only you can make. Stuff that makes you sigh in resignation after waiting for someone else to make happen so you can enjoy it, and realizing that’s never going to happen so you have to get off the couch and do it yourself

--

Do it the entire time with some exasperation. It’ll be great. Happy is out. “I’m so irritated this isn’t done already, we deserve so much better as a species” with a constipated look on your face is in. Hayao Miyazaki “I’m so done with

... (read more)

What motivations tend to drive the largest effect sizes on humanity?

FWIW, I think questions like "what actually causes globally consequential things to happen or not happen" are one of the areas in which we're most dropping the ball. (AI Impacts has been working on a few related question, more like "why do people sometimes not do the consequential thing?")

How do you control for survivorship bias?

I think it's good to at least spot check and see if there are interesting patterns. If "why is nobody doing X???" is strongly associated with large effects, this seems worth knowing, even if it doesn't constitute a measure of expected effect sizes.

Like, keep your eye out. For sure, keep your eye out.

I think this is related to my relative optimism about people spending time on approaches to alignment that are clearly not adequate on their own. It's not that I'm particularly bullish on the alignment schemes themselves, it's that don't think I'd realized until reading this post that I had been assuming we all understood that we don't know wtf we're doing so the most important thing is that we all keep an eye out for more promising threads (or ways to support the people following those threads, or places where everyone's dropping the ball on being prepared for a miracle, or whatever). Is this... not what's happening?

5Linda Linsefors1y
No by default. I did not have this mindset right away. When I was new to AI Safety I though it would require much more experience before I was qualified to question the consensus, because that is the normal situation, in all the old sciences. I knew AI Safety was young, but I did not understand the implications at first. I needed someone to prompt me to get started.  Because I've run various events and co-founded AI Safety Support, I've talked to loooots of AI Safety newbies. Most people are too causes when it comes to believing themselves and too ready to follow authorities. It's usually only takes a short conversation pointing out how incredibly young AI Safety is, and what that means, but many people do need this one push.

75% of sufferers are affected day to day so its not just a cough for the majority its impacting peoples lives often very severely.

The UK source you link for this month says:

The proportion of people with self-reported long COVID who reported that it reduced their ability to carry out daily activities remained stable compared with previous months; symptoms adversely affected the day-to-day activities of 775,000 people (64% of those with self-reported long COVID), with 232,000 (19%) reporting that their ability to undertake their day-to-day activities ha

... (read more)

I agree that classic style as described by Thomas and Turner is a less moderate and more epistemically dubious way of writing, compared to what Pinker endorses. For example, from chapter 1 of Clear and Simple as the Truth:

Classic style is focused and assured. Its virtues are clarity and simplicity; in a sense so are its vices. It declines to acknowledge ambiguities, unessential qualifications, doubts, or other styles.

...

The style rests on the assumption that it is possible to think disinterestedly, to know the results of disinterested thought, and to pre

... (read more)

One of the reasons I want examples is because I think this post is not a great characterization of the kind of writing endorsed in Sense of Style. Based on this post, I would be somewhat surprised if the author had read the book in any detail, but maybe I misremember things or I am missing something.

[I typed all the quotes in manually while reading my ebook, so there are likely errors]

Self-aware style and signposting

Chapter 1 begins:

"Education is an admirable thing," wrote Oscar Wilde, "but it is well to remember from time to time that nothing that is wo

... (read more)
0Cleo Nardo1y
I agree that Pinker' advice is moderate — e.g. he doesn't prohibit authors from self-reference. But this isn't because classic style is moderate — actually classic style is very strict — e.g. it does prohibit authors from self-reference. Rather, Pinker's advice is moderate because he weakly endorses classic style. His advice is "use classic style except in rare situations where this would be bad on these other metric. If I've read him correctly, then he might agree with all the limitations of classic style I've mentioned. (But maybe I've misread Pinker. Maybe he endorses classic style absolutely but uses "classic style" to refer to a moderate set of rules.)
6habryka1y
Huh, this was a great comment. I had read Sense of Style a while ago, and do share many of the OPs complaints about other writing advice, so I did confabulate that Sense of Style was giving the kind of advice the OP argues against, but this comment has convinced me that I was wrong.

I would find this more compelling if it included examples of classic style writing (especially Pinker's writing) that fail at clear, accurate communication.

Agreed, but I'd also like examples from commenters who disagree with OP, of self-aware style that they consider bad. I wonder if my reaction would be "oh I didn't even notice the things that distracted you so much" or "yeah that seems excessive to me too" or what.

A common generator of doominess is a cluster of views that are something like "AGI is an attractor state that, following current lines of research, you will by default fall into with relatively little warning". And this view generates doominess about timelines, takeoff speed, difficulty of solving alignment, consequences of failing to solve alignment on the first try, and difficulty of coordinating around AI risk. But I'm not sure how it generates or why it should strongly correlate with other doomy views, like:

  1. Pessimism that warning shots will produce
... (read more)

Well said. I might quibble with some of the details but I basically agree that the four you list here should theoretically be only mildly correlated with timelines & takeoff views, and that we should try to test how much the correlation is in practice to determine how much of a general doom factor bias people have.

Montgolfier's balloon was inefficient, cheap, slapped together in a matter of months


I agree the balloons were cheap in the sense that they were made by a couple hobbyists. It's not obvious to me how many people at the time had the resources to make one, though.

As for why nobody did it earlier, I suspect that textile prices were a big part of it. Without doing a very deep search, I did find a not-obviously-unreliable page with prices of things in Medieval Europe, and it looks like enough silk to make a balloon would have been very expensive. A sphere with a... (read more)

it's still not the case that we can train a straightforward neural net on winning and losing chess moves and have it generate winning moves. For AlphaGo, the Monte Carlo Tree Search was a major component of its architecture, and then any of the followup-systems was trained by pure self-play.

AlphaGo without the MCTS was still pretty strong:

We also assessed variants of AlphaGo that evaluated positions using just the value network (λ = 0) or just rollouts (λ = 1) (see Fig. 4b). Even without rollouts AlphaGo exceeded the performance of all other Go programs, d

... (read more)
2habryka1y
Yeah, I mean, to be clear, I do definitely think you can train a neural network to somehow play chess via nothing but classification. I am not sure whether you could do it with a feed forward neural network, and it's a bit unclear to me whether the neural networks from the 50s are the same thing as the neural networks from 2000s, but it does sure seem like you can just throw a magic category absorber at chess and then have it play OK chess. My guess is modern networks are not meaningfully more complicated, and the difference to back then was indeed just scale and a few tweaks, but I am not super confident and haven't looked much into the history here.

Here's a selection of notes I wrote while reading this (in some cases substantially expanded with explanation).

The reason any kind of ‘goal-directedness’ is incentivised in AI systems is that then the system can be given an objective by someone hoping to use their cognitive labor, and the system will make that objective happen. Whereas a similar non-agentic AI system might still do almost the same cognitive labor, but require an agent (such as a person) to look at the objective and decide what should be done to achieve it, then ask the system for that. G

... (read more)

"Paxlovid's usefulness is questionable and could lead to resistance. I would follow the meds and supplements suggested by FLCC"

Their guide says:

In a follow up post-marketing study, Paxlovid proved to be ineffective in patients less than 65 years of age and in those who were vaccinated.

This is wrong. The study reports the following:

Among the 66,394 eligible patients 40 to 64 years of age, 1,435 were treated with nirmatrelvir. Hospitalizations due to Covid-19 occurred in 9 treated and 334 untreated patients: adjusted HR 0.78 (95% CI, 0.40 to 1.53). Death due

... (read more)

I was going to complain that the language quoted from the abstract in the frog paper is sufficiently couched that it's not clear the researchers thought they were measuring anything at all. Saying that X "suggests" Y "may be explained, at least partially" by Z seems reasonable to me (as you said, they had at least not ruled out that Z causes Y). Then I clicked through the link and saw the title of the paper making the unambiguous assertion that Z influences Y.

When thinking about a physics problem or physical process or device, I track which constraints are most important at each step. This includes generic constraints taught in physics classes like conservation laws, as well as things like "the heat has to go somewhere" or "the thing isn't falling over, so the net torque on it must be small".

Another thing I track is what everything means in real, physical terms. If there's a magnetic field, that usually means there's an electric current or permanent magnet somewhere. If there's a huge magnetic field, that usual... (read more)

Communication as a constraint (along with transportation as a constraint), strikes me as important, but it seems like this pushes the question to "Why didn't anyone figure out how to control something that's more than a couple weeks away by courier?"

I suspect that, as Gwern suggests, making copies of oneself is sufficient to solve this, at least for a major outlier like Napoleon. So maybe another version of the answer is something like "Nobody solved the principle-agent problem well enough to get by on communication slower than a couple weeks". But it stil... (read more)

in a slow takeoff world, many aspects of the AI alignment problems will already have showed up as alignment problems in non-AGI, non-x-risk-causing systems; in that world, there will be lots of industrial work on various aspects of the alignment problem, and so EAs now should think of themselves as trying to look ahead and figure out which margins of the alignment problem aren’t going to be taken care of by default, and try to figure out how to help out there.

I agree with this, and I think it extends beyond what you're describing here. In a slow takeoff... (read more)

I agree. We're working on making it more browsable and making the hierarchical structure more apparent.

It is! You can see the announcement here.

7hath2y
Thank you! Out of curiosity, are you accepting under-18s for the internship, or for research assistant positions?

Interesting, thanks for sharing.

I don't think I'd take the paper itself seriously. Melatonin Research doesn't seem to be a real journal and the paper looks very unprofessional. One of the paper's authors is a lighting engineer who I think is trying to get people to use indoor lighting with more NIR, while the other seems to be a late-career melatonin guy who probably did some good research at some point. This isn't to say that we should dismiss things unless they're written by Credentialed Scientists® in a Serious Journal® and formatted in The Right Way®. ... (read more)

-1sooh2y
Thanks for taking the time to read. Here's a 4 time board certified MD's primer on light that mentions the paper (and related research):  Other highlights: * Sun exposure decreases all cause mortality, including from melanoma * Sun exposure has beneficial effect on brain volume in multiple sclerosis, independent of consequent vit D

I had sort of vaguely assume you were already doing something like this. It is pretty close to what I used to do for assigning grades while avoiding a "barely missed out" dynamic, in which someone would miss the cutoff for an A by 0.25%.

Load More