focused research organizations (FROs) were conceived (by Convergent et al) as "startup nonprofits" on a deadline, they're planned to exist for 4-7 years or so to fight scope creep for specific missions.
One relatively obvious objection is with the world changing so much, how can you plan 5 years of "focus" up front? 5 years may be too long to successfully articulate a focus that makes sense for even that long.
I would propose microFROs, ~90-200 day sprints in the spirit of a FRO that spin up, kick ass, and spin down.
Some generalizable piece of advice from HOW TO SUCCEED IN MRBEAST PRODUCTION:
...USE CONSULTANTS
Consultants are literally cheat codes. Need to make the world's largest slice of cake? Start off by calling the person who made the previous world’s largest slice of cake lol. He’s already done countless tests and can save you weeks worth of work. I really want to drill this point home because I’m a massive believer in consultants. Because I've spent almost a decade of my life hyper obsessing over youtube, I can show a brand new creator how to go from 100 subscriber
If "a (woman|man) is someone with a deeply felt sense that they are (female sex|male sex)," is the definition people are using, that would predict hormone replacement therapies, sexual reassignment surgeries, and a reported mismatch between someone's gender and their sex. That usage seems to fit what we observe really well.
I'm eager to know if this is missing something. The next most likely thing people could mean, I think, is "A (woman|man) is someone with a (female brain|male brain)."
I see I've skipped a step in the original comment.
I mean that the definition predicts the behavior because it predicts the felt sense. If someone says to me, "I am a man," and I think they are using the word to mean "a deeply felt sense of being male," I'd expect them to have a deeply felt sense that they're male. Because they have said sense, I'd then expect a desire for HRT if they aren't male.
So "I am a man," predicts the male-sex sense which predicts HRT, if both the the definition of "man" is "male-sex sense" and the speaker is female.
The internal sen...
Summary of why I don't buy diachronic Dutch book arguments
(Using Elga's Dutch book against imprecise credences as an example, where the agent faces a sequence of two bets A and B.)
I agree with Claude's 4th point.
Are there existing posts on how/whether deployment of AI in enterprise is a threat vector for catastrophic risks? E.g.:
- What can a power-seeking AI in a large consultancy/accountancy firm do, given it will have access to 1000s of other large businesses data/systems?
- Which catastrophic risks have paths that go through enterprise deployments?
- What norms/best practices/governance can enterprise follow to reduce catastrophic risks, if any?
- Is this a worthwhile lever for the AI Safety community to act on? E.g. somehow doing outreach to people in enterprise....
Why even the highly RL-learned LM agents do not seem to follow the substrate-upgrading and resource-hoarding sub-goals?
1. They are not utility maximizers, but rather target satisfiers, goal achievers.
2. The goal horizon is not far enough to be worth pursuing these potential-improving long-term subgoals.
In my opinion (feel free to ignore it), it makes sense to create a tag if you have already at least three examples of things that could be tagged by it. A tag is a way to group existing content, not an way to create it.
As a nerd who considers myself part of many nerd spaces, I am ashamed and my conscience is disturbed by how unwelcoming all kinds of nerd spaces tend to be towards women. Wikipedia editors, for example. They're definitely nerds. They stick to strict checklists about style and word choice, strict checklists about article structure and layout, strict checklists about what is and isn't notable (i.e. worthy of being made into an article). They should be immune to this, or at least highly resistant. And yet scholarly articles and studies are written showing tha...
Do you have a specific falsifiable theory on how specifically the nerd places are unwelcoming towards women? Without a specific theory, it is not obvious what specifically could be done. Pointless self-flagellation is not helping anyone.
One possibility is that women are simply statistically less interested in these kinds of activities. I am not saying that this is definitely the full answer. But it's an example of a possible cause that has no one to blame.
I am a man and I feel unwelcome at Wikipedia recently. (I wrote about it repeatedly on ACX, but I can'...
Epistemic status: not sure.
People comparing Mackenzie Scott's charitable donations to EA charities to dunk on them for "being ineffective" on a health basis seem to be arguing in pretty bad faith.
A large portion of those donations weren't health-focused at all, but rather cause-focused. This doesn't seem more out of line to me than (say) EA focus on x-risk non-profits: Both have nebulous return in a quantifiable sense, but both do (for better or worse) make noticeable impacts in the world on a long enough timescale.
Sometimes a person cares about a thing th...
I think Eliezer Yudkowsky and everyone else who were around in the old days have made a huge counterfactual impact by entangling AI safety with rationalism and good epistemics. I can easily imagine an alternative world where AI safety is still picked up, but like many movements, the passionate, smart people are attracted into unproductive directions because they stick to what's memetic, avoid conflicting with allies, and are too panicked to think carefully.
Sometimes people ask me if AI safety would be better off if we did nothing and never brought attention to AGI, and this is one of the primary reasons I think that is not true.
Some subcultures actually have done good, so the base rate is not zero. In fact, most groups with an explicit purpose end up advancing that purpose, so the base rate is 50% or more. Whether the purpose is a good one is more fraught. But even there we must evaluate cases and details, rather than just saying "most people believe self-serving things, so you're doing that too."
On Plan A vs. Plan S
Plan A scales to top-expert-level AI in ~5 years, then to superintelligence once very high confidence in alignment is achieved; in the scenario this is 5 more years later, for a total of 10 years from the deal beginning until superintelligence. (More on the scaling strategy in https://ai-2040.com/supplements/capability-scaling-strategy.)
The version of Plan S that we’re most sympathetic to involves a halt on AI capabilities for a minimum of ~3-5 years with intent to eventually scale to top-expert-level AI and then superintelligence, but ...
1I think it's much easier to achieve Plan S than Plan A.
Plan S is simple "Don't train new AI until there's very strong consensus it's a good idea."
Plan A requires continuously making nuanced judgment calls about what counts as safe, while generally maintaining momentum on training more powerful AIs (and leaving lots of dry tinder around). It's essentially an unsolved problem to make regulations careful enough to distinguish good vs bad safety cases, and I don't think the Plan A documents had particularly good ideas for now to do so.
I think Plan A basicall...
1Homework zero in my AI safety course is now online
I was thinking about weaknesses of the control agenda, which matters because plans like Plan A rely on control as a defense when scaling to superintelligence. I think it's important we mitigate these weaknesses before we rely on control for safety, its important we don't overestimate safety when relying on control evals. Confidence: medium (I think others have thought through these issues as well and have better solutions).
I think people on LW somewhat overestimate winning over persistence, especially for generating new perspectives and ideas on deep issues.
If we look at AI Safety to now as a pre-paradigmatic field then one of the main ways of progress is finding frames and good questions to ask. What determines the degree to which a field is good at asking questions? Partly the independently distributed information it can gain, partly the actionability of that information.
Now, what does that mean for you as a theorist? Well, you should have something that 1. brings new use...
1As Plan A was coming together, I made this diagram to explain to the team why Total Research Transparency seemed so important to me, and why transparency more broadly did. For example, it's very important for preventing concentration of power. (Explanation below)
First, what's going on? The green boxes are the goals we are trying to achieve, and the blue boxes are the main interventions we are recommending. (As the scenario makes clear, there are more interventions and goals besides these, but the diagram is complicated enough as it is, so they won't be men...
2Both boxes are blue, meaning they are both key interventions we recommend. While one helps enable the other, we don't think doing one is sufficient to cause the other. In addition to the transparency we also recommend the limits.
1I've been spending a few hours a month tutoring high-school-level maths, and being reminded of exactly how janky our standard notation is. Putting two numbers next to each other means "multiply these together" unless they're parts of a mixed fraction; raising something to the power of -1 can mean "invert this function" or "take the reciprocal of this"[1]; all our variable names have implicit meanings almost no-one bothers to spell out.
This is especially egregious w.r.t. trig functions: cos^-1 of x multiplied by cos^3 of x is NOT cos^2 of x!
Often when I give advice I do it as an anecdote. This especially applies to:
Instead of suggesting what someone else should do, I tell a story of a similar problem I had and solved, and then let the other person pull out what ever part of the lesson applies to them. I find this works better for a number of reasons.
1This is the best and most useful Quick Take I can remember reading.
1
