When we were looking for office space in Berkeley earlier this year we were seeing list price between $3.25-$2.75/month per square foot, or $780k-900k/year for 20,000 square feet. I'd expect with negotiation you could get somewhat better pricing than this implies, especially if committing to a longer time period.
Yep, if you commit for longer time periods you can definitely get better deals, and there are definitely other ways to save on office space costs. I didn't mean to imply this was the minimum you could rent office space for.
The $1.2M/yr estimate was...
With that in mind, I was surprised by the lack of information in this funding request. I feel mixed about this: high-status AIS orgs often (accurately) recognize that they don't really need to spend time justifying their funding requests, but I think this often harms community epistemics (e.g., by leading to situations where everyone is like "oh X org is great-- I totally support them" without actually knowing much about what work they're planning to do, what models they have, etc.)
Sorry about that! I've drafted like 3-4 different fundraising posts over th...
Will much of that $3-6M go into renovating and managing the Rose Garden Inn, or to cover work that could have been covered by existing funding if the Inn wasn't purchased?
Thinking about the exact financing of the Inn is a bit messy, especially if we compare it to doing something like running the Lightcone Offices, because of stuff like property appreciation, rental income from people hosting events here, and the hard-to-quantify costs of tying up capital in real estate as opposed to more liquid assets like stocks.
If you assume something like 5% property ap...
We are just wrapping up renovations so not much yet (though we are done very soon). This summer we are likely hosting a good chunk of the SERI MATS scholars, as well as providing space for various other retreats and events (like the Singular Value Learning Theory workshop and we are talking to Manifold about maybe running a 100+ person forecasting conference here).
In-parallel we are also providing office space to a small number of people that I expect to slowly grow over time, trying to build a tight-knit community of people working to reduce existen...
Those names do seem like at least a bit of an update for me.
I really wish that having someone EA/AI-Alignment affiliated who has expressed some concern about x-risk was a reliable signal that a project will not end up primarily accelerationist, but alas, history has really hammered it in for me that that is not reliably true.
Some stories that seem compatible with all the observations I am seeing:
Agreed, the initial announcement read like AI safety washing and more political action is needed, hence the call to action to improve this.
But read the taskforce leader’s op-ed:
I am confused why you are framing this in a positive way? The announcement seems to primarily be that the UK is investing $125M into scaling AI systems in order to join a global arms race to be among the first to gain access to very powerful AI systems.
The usage of "safety" in the article seems to have little to do with existential risk, and indeed seems mostly straightforward safety-washing.
Like, I am genuinely open to this being a good development, and I think a lot of recent development around AI policy and the world's relationship to AI risk has been good, but I do really have trouble seeing how this announcement is a positive sign.
If you get funding from other funds, it would be best if you update your application (you can edit your application any time before the evaluation period ends), or withdraw your application. We'll get notifications if you make edits and make sure to consider them.
Yep, just paying a person a salary works, though the person needs to do enough things that are somewhat legibly for the public benefit to justify their salary to the IRS.
In an environment where EA organizations don't seem to keep their promises about responding in promised timeframes and write things like "we get back to you" and then don't and seem to be willing to accept the negative mental health consequences that come along with that, is there a good reason why people should expect a different practice from this new process?
Yeah, it's a pretty fair criticism. I am quite confident we will keep the "responses around the start of August" mark, because that one is pretty inherently baked into our evaluation process. And I ...
You have a clause about China and India, but not about Russia. So, Russia is OK? (Among other things, in Russia, it is difficult to receive money from abroad: many banks are disconnected from SWIFT, some of the rest have stopped working with transactions from the US on their own initiative, and there is a chance that a Western bank will refuse to conduct a transaction to Russia. So the most reliable way is to have a trusted intermediary person with money in a bank account in Russia and a second bank account somewhere else.)
I think Russia is marginally more...
Lightspeed Grants is definitely meaningfully modeled as being a kind of spinoff of the SFF, and also as a way to create more competition between different funding distribution mechanisms for Jaan and other funders.
This means for this round there are a lot of similarities on the backend, though I do expect the applicant experience to already be quite different. And then I expect much more heavy divergence in future rounds as we have more end-to-end ownership over the product, which allows us to make more changes (I've already made a lot of changes to ...
Yeah, I do think there are a bunch of benefits to doing things in Google Docs, though it is often quite useful to have more structured data on the evaluation side.
This increases re-usability and decreases stress, as it's easy to make updates later on so it's less of a worry that you ended up missing something crucial.
You can actually update your application any time. When you submit the application you get a link that allows you to edit your submission as well as see its current status in the evaluation process. Seems like we should sign-post this better.
Oh, I actually like the Lightspeed Grants logo more.
It's the Lightcone Logo with a dollar sign in it!
Intros would be great! Now that we've launched I've been planning to reach out to more potential funders, and I think we will very likely get more good applications than we have funding for.
Feel free to send me a DM or send me an email at habryka@lightspeedgrants.org to coordinate.
It is kind of a logistical headache to handle withdrawn application after we figured out a funding allocation, though it's not that bad.
If you do have a lot of uncertainty on whether you will actually want to go ahead with the project (or think it's somewhat conditional on funder enthusiasm), I think it's best to choose the "get a response within 2 weeks" option. That's I think also the best option if you are applying for multiple projects (in which case I would recommend filling out one application that gets processed in the 60-day window, and then some secondary applications that you might pivot to if you get funding within the 2 week window).
Might we perhaps refrain from dismissing it if we can't even remember what the prior proposals were?
I mean, I definitely remember! I could summarize them, I just don't have a link ready, since they were mostly in random comment threads. I might go through the effort of trying to search for things, but the problem is not one of remembering, but one of finding things in a see of 10 years of online discussion in which many different terms have been used to point to the relevant ideas.
...The linked post argues that this has important safety implications. So point
With all due respect to Gwern, repeating claims that work has already been done and then refusing to substantiate them is an epistemic train wreck.
I don't think that's what's happening here, so I feel confused about this comment. I haven't seen Gwern 'refuse to substantiate them'. He indeed commented pretty extensively about the details of your comment.
Shutdown-seekingness has definitely been discussed a bunch over the years. It seems to come up a lot in Tool-AI adjacent discussions as well as impact measures. I also don't have a great link here sadl...
Reacts can also be downvoted, which results in them being hidden. This is to counter abuse in the same way as voting counters abuse via low-quality comments.
Cool, makes sense. Sounds like I remembered the upper bound for the algorithmic efficiency estimate. Thanks for correcting!
You just copy the link to the market, and if you paste it into an empty new paragraph it should automatically be replaced with an embed.
I think requiring a "common initialization + early training trajectory" is a pretty huge obstacle to knowledge sharing, and would de-facto make knowledge sharing among the vast majority of large language models infeasible.
I do think stuff like stitching via cross-attention is kind of interesting, but it feels like a non-scalable way of knowledge sharing, unless I am misunderstanding how it works. I don't know much about Knowledge Distillation, so maybe that is actually something that would fit the "knowledge sharing is easy" description (my models he...
I think requiring a "common initialization + early training trajectory" is a pretty huge obstacle to knowledge sharing, and would de-facto make knowledge sharing among the vast majority of large language models infeasible.
Agreed. That part of my comment was aimed only at the claim about weight averaging only working for diffusion/image models, not about knowledge sharing more generally.
...I do think stuff like stitching via cross-attention is kind of interesting, but it feels like a non-scalable way of knowledge sharing, unless I am misunderstanding how
Huh, maybe. My current guess is that things aren't really "compute bottlenecked". It's just the case that we now have profitable enough AI that we really want to have better compute. But if we didn't get cheaper compute, we would still see performance increase a lot as we find ways to improve compute-efficiency the same way we've been improving it a lot over the past 5-10 years, and that for any given period of time, the algorithmic progress is a bigger deal for increasing performance than the degree to which compute got cheaper in the same period.
If we take this as the disagreement -- will AI progress come from a handful of big insights, or many small ones -- I think the world right looks a great deal more like Hanson's view than Yudkowsky's. In his interview with Lex Fridman, Sam Altman characterizes GPT-4 as improving on GPT-3 in a hundred little things rather than a few big things, and that's... by far... my impression of current ML progress. So when I interpret their disagreement in terms of the kind of work you need to do before attaining AGI, I tend to agree that Hanson is right.
This also fee...
I agree I'm confused here. But it's hard to come down to clear interpretations. I kinda think Hanson and Yudkowsky are also confused.
Like, here are some possible interpretations on this issue, and how I'd position Hanson and Yudkowsky on them based on my recollection and on vibes.
But -- regardless of Yudkowsky's current position -- it still remains that you'd have been extremely surprised by the last decade's use of compute if you had believed him, and much less surprised if you had believed Hanson.
I think you are pointing towards something real here, but also, algorithmic progress is currently outpacing compute growth by quite a bit, at least according to the Epoch AI estimates I remember. I also expect algorithmic progress to increase in importance.
I do think that some of the deep learning revolution turned out to be kind o...
I do think that some of the deep learning revolution turned out to be kind of compute bottlenecked, but I don't believe this is currently that true anymore
I had kind of the exact opposite impression of compute bottlenecks (that deep learning was not meaingfully compute bottlenecked until very recently). OpenAI apparently has a bunch of products and probably also experiments that are literally just waiting for H100s to arrive. Probably this is mainly due to the massive demand for inference, but still, this seems like a kind actual hardware bottleneck that i...
An actual improvement to say, how Transformers work, would help with speech recognition, language modelling, image recognition, image segmentation, and so on and so forth. Improvements to AI-relevant hardware are a trillion-dollar business. Work compounds so easily on other work that many alignment-concerned people want to conduct all AI research in secret.
This section feels like it misunderstands what Yudkowsky is trying to say here, though I am not confident. I expected this point to not be about "what happens if you find an improvement to transformers i...
...Moreover, granting neural networks, trading cognitive content has turned out to be not particularly hard. It does not require superintelligence to share representations between different neural networks; a language model can be adapted to handle visual data without enormous difficulty. Encodings from BERT or an ImageNet model can be applied to a variety of downstream tasks, and this is by now a standard element in toolkits and workflows. When you share architectures and training data, as for two differently fine-tuned diffusion models, you can get semantic
I would also call this one for Eliezer. I think we mostly just retrain AI systems without reusing anything. I think that's what you'd guess on Eliezer's model, and very surprising on Robin's model. The extent to which we throw things away is surprising even to a very simple common-sense observer.
I would have called "Human content is unimportant" for Robin---it seems like the existing ML systems that are driving current excitement (and are closest to being useful) lean extremely heavily on imitation of human experts and mostly don't make new knowledge thems...
In addition to what cfoster0 said, I'm kinda excited about the next ~2-3 years of cross LLM knowledge transfer, so this seems a differing prediction about the future, which is fun.
My model for why it hasn't happened already is in part just that most models know the same stuff, because they're trained on extremely similar enormous swathes of text, so there's no gain to be had by sticking them together. That would be why more effort goes into LLM / images / video glue than LLM / LLM glue.
But abstractly, a world where LLMs can meaningfully be connected to vi...
The part where you can average weights is unique to diffusion models, as far as I can tell, which makes sense because the 2-d structure of the images is very local, and so this establishes a strong preferred basis for the representations of different networks.
...Exchanging knowledge between two language models currently seems approximately impossible? Like, you can train on the outputs, but I don't think there is really any way for two language models to learn from each other by exchanging any kind of cognitive content, or to improve the internal represen
This feels like it is not really understanding my point, though maybe best to move this to some higher-bandwidth medium if the point is that hard to get across.
Giving it one last try: What I am saying is that I don't think "conventional notion of preferences" is a particularly well-defined concept, and neither are a lot of other concepts you are using in order to make your predictions here. What it means to care about the preferences of others is a thing with a lot of really messy details that tend to blow up in different ways when you think harder a...
I think some of the confusion here comes from my using "kind" to refer to "respecting the preferences of existing weak agents," I don't have a better handle but could have just used a made up word.
Yeah, sorry, I noticed the same thing a few minutes ago, that I was probably at least somewhat misled by the more standard meaning of kindness.
Tabooing "kindness" I am saying something like:
Yes, I don't think extrapolated current humans assign approximately any value to the exact preference of "respecting the preferences of existing weak agents" and I...
...If the result of an optimization process will be predictably horrifying to the agents which are applying that optimization process to themselves, then they will simply not do so.
In other words: AIs which feel anything in the vicinity of kindness before applying cosmic amounts of optimization pressure to themselves will try to steer that optimization pressure towards something which is recognizably kind at the end.
And I don't think there's any good argument for why AIs will lack any scrap of kindness with very high confidence at the point where they're just
Meta: I feel pretty annoyed by the phenomenon of which this current conversation is an instance, because when people keep saying things that I strongly disagree with which will be taken as representing a movement that I'm associated with, the high-integrity (and possibly also strategically optimal) thing to do is to publicly repudiate those claims*, which seems like a bad outcome for everyone.
For what it's worth, I think you should just say that you disagree with it? I don't really understand why this would be a "bad outcome for everyone". Just list out th...
Humans might respect the preferences of weak agents right now, but if they thought about it for longer they'd pretty robustly just want to completely destroy the existing agents (including a hypothetical alien creator) and replace them with something better. No reason to honor that kind of arbitrary path dependence.
No, this doesn't feel accurate. What I am saying is more something like:
The way humans think about the question of "preferences for weak agents" and "kindness" feels like the kind of thing that will come apart under extreme optimization, i...
Might write a longer reply at some point, but the reason why I don't expect "kindness" in AIs (as you define it here) is that I don't expect "kindness" to be the kind of concept that is robust to cosmic levels of optimization pressure applied to it, and I expect will instead come apart when you apply various reflective principles and eliminate any status-quo bias, even if it exists in an AI mind (and I also think it is quite plausible that it is completely absent).
Like, different versions of kindness might or might not put almost all of their conside...
If the result of an optimization process will be predictably horrifying to the agents which are applying that optimization process to themselves, then they will simply not do so.
In other words: AIs which feel anything in the vicinity of kindness before applying cosmic amounts of optimization pressure to themselves will try to steer that optimization pressure towards something which is recognizably kind at the end.
And I don't think there's any good argument for why AIs will lack any scrap of kindness with very high confidence at the point where they're just...
Is this a fair summary?
Humans might respect the preferences of weak agents right now, but if they thought about it for longer they'd pretty robustly just want to completely destroy the existing agents (including a hypothetical alien creator) and replace them with something better. No reason to honor that kind of arbitrary path dependence.
If so, it seems like you wouldn't be making an argument about AI or aliens at all, but rather an empirical claim about what would happen if humans were to think for a long time (and become more the people we wished to be a...
Almost certainly related to that email controversy from a few months ago. My sense is people have told him (or he has himself decided) to take a step back from public engagement.
I think I disagree with this, but it's not a totally crazy call, IMO.
This feels game-theoretically pretty bad to me, and not only abstractly, but I expect concretely that setting up this incentive will cause a bunch of people to attempt to go into capabilities (based on conversations I've had in the space).
We had an earlier iteration of the design where each react was basically a dimension where it made sense to have positives and negatives, and it IMO constrained the space of reacts too much.
The primary point of the anti-react system is as a corrective system that I expect to be used relatively rarely (but that I do think is important to exist). While I agree that some reactions have meaningful opposites that one might be tempted to express with an anti-react, the right thing to do IMO is to provide another react with the opposite meaning, so that you can see them both side-by-side.
I think I changed my mind on this, FWIW, after playing around more in this thread. I think bottom-left is indeed better.
I don't think it means anything (it's also not a particularly accessible state for the UI to end up in, since you have to first react, and then anti-react). It is kind of a state that's hard to avoid, where maybe you anti-reacted to something after someone left a react, and then they withdraw it, in which case it seems bad to just throw away the information that you thought it was a bad react, and seems more appropriate to "apply" it whenever the next person were to leave the same react again.
Currently the most-reacted to icons are on the very right of the list. This feels like it's the wrong way around. I want to notice the most-reacted icons first, which will still be on the left side.
Bug: I can't undo my vote in the react/anti-react voting widget. When I click on the upvote/downvote buttons it just reapplies the vote instead of undoing it.
Having some way to view whether I've already left a reaction on a post would be great. Currently it just shows a number, and then if I click on it, the number decreases if I've already left a reaction. Would be nice for the background to be some color if I left the relevant reaction (maybe green if I pro-reacted and red if I anti-reacted).
Mod note: It felt fine to do this once or twice, but it's not an intended use-case of AI Alignment Forum membership to post to the AI Alignment Forum with content that you didn't write.
I would have likely accepted this submission to the AI Alignment Forum anyways, so it seems best to just go via the usual submission channels. I don't want to set a precedent of weirdly confusing co-authorship for submission purposes. You can also ping me on Intercom in-advance if you want to get an ahead notice of whether the post fits on the AIAF, or want to make sure it goes live there immediately.
I did sure expect more text for this question. Is there any specific reason for why it should have worked out better than it did? Seems like it was pretty great for a lot of it, but wasn't enough to fully close the gap between them and the rest of Europe. Is the question why they didn't fully close the gap?
Mod note: I removed Dan H as a co-author since it seems like that was more used as convenience for posting it to the AI Alignment Forum. Let me know if you want me to revert.
Huh, interesting. Seems good to get an HTML version then, since in my experience PDFs have a pretty sharp dropoff in readership.
When I google the title of the paper literally the only hit is this LessWrong post. Do you know where the paper was posted and whether there exists an HTML version (or a LaTeX, or a Word, or a Google Doc version)?
If the difference between these papers is: we do activations, they do weights, then I think that warrants more conceptual and empirical comparisons.
Yeah, it's totally possible that, as I said, there is a specific other paper that is important to mention or where the existing comparison seems inaccurate. This seems quite different from a generic "please have more thorough related work sections" request like the one you make in the top-level comment (which my guess is was mostly based on your misreading of the post and thinking the related work section only spans two paragraphs).
The level of comparison between the present paper and this paper seems about the same as I see in papers you have been a co-author in.
E.g. in https://arxiv.org/pdf/2304.03279.pdf the Related Works section is basically just a list of papers, with maybe half a sentence describing their relation to the paper. This seems normal and fine, and I don't see even papers you are a co-author on doing something substantively different here (this is again separate from whether there are any important papers omitted from the list of related works, or whether any s...
I don't understand this comment. I did a quick count of related works that are mentioned in the "Related Works" section (and the footnotes of that section) and got around 10 works, so seems like this is meeting your pretty arbitrarily established bar, and there are also lots of footnotes and references to related work sprinkled all over the post, which seems like the better place to discuss related work anyways.
I am not familiar enough with the literature to know whether this post is omitting any crucial pieces of related work, but the relevant section of ...
At least for the coming year, our expenses are pretty entangled between all the different projects in a way that makes differentially funding things hard. I do take preferences of our donors on how to focus our efforts into account, so donating and just telling us that you would prefer us to work more on one kind of thing vs. another will have some effect.
My guess is you will mostly just have to average our impact across different areas and decide whether the whole portfolio is above your bar.