by Quinn1 min read16th Jan 202175 comments
New Comment
74 comments, sorted by Click to highlight new comments since: Today at 2:10 PM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Good arguments - notes on Craft of Research chapter 7

Arguments take place in 5 parts.

  1. Claim: What do you want me to believe?
  2. Reasons: Why should I agree?
  3. Evidence: How do you know? Can you back it up?
  4. Acknowledgment and Response: But what about ... ?
  5. Warrant: How does that follow?

This can be modeled as a conversation with readers, where the reader prompts the writer to taking the next step on the list.

Claim ought to be supported with reasons. Reasons ought to be based on evidence. Arguments are recursive: a part of an argument is an acknowledgment of an anticipated response, and another argument addresses that response. Finally, when the distance between a claim and a reason grows large, we draw connections with something called warrants.

The logic of warrants proceeds in generalities and instances. A general circumstance predictably leads to a general consequence, and if you have an instance of the circumstance you can infer an instance of the consequence.

Arguing in real life papers is complexified from the 5 steps, because

  • Claims should be supported by two or more reasons
  • A writer can anticipate and address numerous responses. As I mentioned, arguments are recursive, especi
... (read more)

thoughts on chapter 9 of Craft of Research

Getting the easy things right shows respect for your readers and is the best training for dealing with the hard things.

If they don't believe the evidence, they'll reject the reasons and, with them, your claim.

We saw previously that claims ought to be supported with reasons, and reasons ought to be based on evidence. Now we will look closer at reasons and evidence.

Reasons must be in a clear, logical order. Atomically, readers need to buy each of your reasons, but compositionally they need to buy your logic. Storyboarding is a useful technique for arranging reasons into a logical order: physical arrangements of index cards, or some DAG-like syntax. Here, you can list evidence you have for each reason or, if you're speculating, list the kind of evidence you would need.

When storyboarding, you want to read out the top level reasons as a composite entity without looking at the details (evidence), because you want to make sure the high-level logic makes sense.

Readers will not accept a reason until they see it anchored in what they consider to be a bedrock of established fact. ... To count as evidence, a statement must report something tha

... (read more)

Sources - notes on Craft of Research chapters 5 and 6

Primary, secondary, and tertiary sources

Primary sources provide you with the "raw data" or evidence you will use to develop, test, and ultimately justify your hypothesis or claim. Secondary sources are books, articles, or reports that are based on primary sources and are intended for scholarly or professional audiences. Tertiary sources are books and articles that synthesize and report on secondary sources for general readers, such as textbooks, articles in encyclopedias, and articles in mass-circulation publications.

The distinction between primary and secondary sources comes from 19th century historians, and the idea of tertiary sources came later. The boundaries can be fuzzy, and are certainly dependent on the task at hand.

I want to reason about what these distinctions look like in the alignment community, and whether or not they're important.

The rest of chapter five is about how to use libraries and information technologies, and evaluating sources for relevance and reliability.

Chapter 6 starts off with the kind of thing you should be looking for while you read

Look for creative agreement

  • Offer additional support. You can
... (read more)

Yesterday I quit my job for direct work on epistemic public goods! Day one of direct work trial offer is April 4th, and it'll take 6 weeks after that to know if I'm a fulltime hire.

I'm turning down

  • raise to 200k/yr usd
  • building lots of skills and career capital that would give me immense job security in worlds where investment into one particular blockchain doesn't go entirely to zero
  • having fun on the technical challenges


  • confluence of my skillset and a theory of change that could pay huge dividends in the epistemic public goods space
  • 0.35x paycut
... (read more)

nonprosaic ai will not be on short timelines

I think a property of my theory of change is that academic and commercial speed is a bottleneck. I recently realized that my mass assignment for timelines synchronized with my mass assignment for the prosaic/nonprosaic axis. The basic idea is that let's say a radical new paper that blows up and supplants the entire optimization literature gets pushed to the arxiv tomorrow, signaling the start of some paradigm that we would call nonprosaic. The lag time for academics and industry to figure out what's going on, fi... (read more)

The reasoning assumes that ideas are first generated in academia and don't arise inside of companies. With DeepMind outperforming the academic protein folding community when protein folding isn't even the main focus of DeepMind I consider it plausible that new approaches arise within a company and get only released publically when they are strong enough to have an effect. Even if there's a paper most radical new papers get ignored by most people and it might be that in the beginning only one company takes the idea seriously and doesn't talk about it publically to keep a competive edge.
That's totally fair, but I have a wild guess that the pipeline from google brain to google products is pretty nontrivial to traverse, and not wholly unlike the pipeline from arxiv to product.
2Steven Byrnes1y
How short is "short" for you? Like, AlexNet was 2012, DeepMind patented deep Q learning in 2014, the first TensorFlow release was 2015, the first PyTorch release was 2016, the first TPU was 2016, and by 2019 we had billion-parameter GPT-2 … So if you say "Short is ≤2 years", then yeah, I agree. If you say "Short is ≤8 years", I think I'd disagree, I think 8 years might be plenty for a non-prosaic approach. (I think there are a lot of people for whom AGI in 15-20 years still counts as "short timelines". Depends on who you're talking to, I guess.)
I should've mentioned in OP but I was lowkey thinking upper bound on "short" would be 10 years. I think developer ecosystems are incredibly slow (longer than ten years for a new PL to gain penetration, for instance). I guess under a singleton "one company drives TAI on its own" scenario this doesn't matter, because tooling tailored for a few teams internal to the same company is enough which can move faster than a proper developer ecosystem. But under a CAIS-like scenario there would need to be a mature developer ecosystem, so that there could be competition.

I feel like 7 years from AlexNet to the world of PyTorch, TPUs, tons of ML MOOCs, billion-parameter models, etc. is strong evidence against what you're saying, right? Or were deep neural nets already a big and hot and active ecosystem even before AlexNet, more than I realize? (I wasn't paying attention at the time.)

Moreover, even if not all the infrastructure of deep neural nets transfers to a new family of ML algorithms, much of it will. For example, the building up of people and money in ML, the building up of GPU / ASIC servers and the tools to use them, the normalization of the idea that it’s reasonable to invest millions of dollars to train one model and to fab ASICs tailored to a particular ML algorithm, the proliferation of expertise related to parallelization and hardware-acceleration, etc. So if it took 7 years from AlexNet to smooth turnkey industrial-scale deep neural nets and billion-parameter models and zillions of people trained to use them, then I think we can guess <7 years to get from a different family of learning algorithms to the analogous situation. Right? Or where do you disagree?

No you're right. I think I'm updating toward thinking there's a region of nonprosaic short-timelines universes. Overall it still seems like that region is relatively much smaller than prosaic short-timelines and nonprosaic long-timelines, though.

Excellence and adequacy

I asked a friend whether I should TA for a codeschool called ${{codeschool}}.

You shouldn't hang around ${{codeschool}}. People at ${{codeschool}} are not pursuing excellence.

A hidden claim there that I would soak up the pursuit of non-excellence by proximity or osmosis isn't what's interesting (though I could see that turning out either way). What's interesting is the value of non-excellence, which I'll call adequacy.

${{codeschool}} in this case is effective and impactful at putting butts in seats at companies, and is thereby re... (read more)

Seems to me that on the market there are very few jobs for the SICP types. The more meta something is, the less of that is needed. If you can design an interactive website, there are thousands of job opportunities for you, because thousands of companies want an interactive website, and somehow they are willing to pay for reinventing the wheel. If you can design a new programming language and write a compiler for it... well, it seems that world already has too many different programming languages, but sure there is a place for maybe a dozen more. The probability of success is very small even if you are a genius. The best opportunity for developers who think too meta is probably to design a new library for an already popular programming language, and hope it becomes popular. The question is how exactly you plan to get paid for that. Probably another problem is that it requires intelligence to recognize intelligence, and it requires expertise to recognize expertise. The SICP type developer seems to most potential employers and most potential colleagues as... just another developer. The company does not see individual output, only team output; it does not matter that your part of code does not contain bugs, if the project as a whole does. You cannot use solutions that are too abstract for your colleagues, or for your managers. Companies value replaceability, because it is less fragile and helps to keep developer salaries lower than they might be otherwise. (In theory, you could have a team full of SICP type developers, which would allow them to work smarter, and yet the company would feel safe. In practice, companies can't recognize this type and don't appreciate it, so this is not going to happen.) Again, probably the best position for a SICP type developer in a company would be to develop some library that the rest of the company would use. That is, a subproject of a limited size that the developer can do alone, so they are not limited in the techniques they us

I find myself, just as a random guy, deeply impressed at the operational competence of airports and hospitals. Any good books about that sort of thing?

It is pretty impressive that they function as well as they do, but seeing how the sausage is made (at least in hospitals) does detract from it quite substantially. You get to see not only how an enormous number of battle hardened processes prevent a lot of lethal screw-ups, but also how also how sometimes the very same processes cause serious and very occasionally lethal screw-ups. It doesn't help that hospitals seem to be universally run with about 90% of the resources they need to function reasonably effectively. This is possibly because there is relentless pressure to cut costs, but if you strip any more out of them then people start to die from obviously preventable failures. So it stabilizes at a point where everything is much more horrible than it could be, but not quite to an obviously lethal extent. As far as your direct question goes, I don't have any good books to recommend.

Rats and EAs should help with the sanity levels in other communities

Consider politics. You should take your political preferences/aesthetics, go to the tribes that are based on them, and help them be more sane. In the politics example, everyone's favorite tribe has failure modes, and it is sort of the responsibility of the clearest-headed members of that tribe to make sure that those failure modes don't become the dominant force of that tribe.

Speaking for myself, having been deeply in an activist tribe before I was a rat/EA, I regret I wasn't there to hel... (read more)

But what if that makes my tribe lose the political battle? I mean, if rationality actually helped win political fights, by the power of evolution we already would have been all born rational...
1. Evolution does not magically get from A to B instantly. 2. Evolution does not necessarily care about X for many values of X. This can include: winning political fights, whether or not nukes are built and many other things.

Claims - thoughts on chapter eight of Craft of Research

Broadly, the two kinds of claims are conceptual and practical.

Conceptual claims ask readers not to ask, but to understand. The flavors of conceptual claim are as follows:

  • Claims of fact or existence
  • Claims of definition and classification
  • Claims of cause and consequence
  • Claims of evaluation or appraisal

There's essentially one flavor of practical claim

  • Claims of action or policy.

If you read between the lines, you might notice that a kind of claim of fact or cause/consequence is that a policy work... (read more)

This may be context-dependent. Different countries probably have different cultural norms. Norms may differ for higher-status and lower-status speakers. Humble speech may impress some people, but others may perceive it as a sign of weakness. Also, is your audience fellow scientists or are you writing a popular science book? (More hedging for the former, less hedging for the latter.)

notes (from a very jr researcher) on alignment training pipeline

Training for alignment research is one part competence (at math, cs, philosophy) and another part having an inside view / gears-level model of the actual problem. Competence can be outsourced to universities and independent study, but inside view / gears-level model of the actual problem requires community support.

A background assumption I'm working with is that training as a longtermist is not always synchronized with legible-to-academia training. It might be the case that jr researchers oug... (read more)

I don't think Critch's saying that the best way to get his attention is through cold emails backed up by credentials. The whole post is about him not using that as a filter to decide who's worth his time but that people should create good technical writing to get attention.
Critch's written somewhere that if you can get into UC Berkeley, he'll automatically allow you to become his student, because getting into UC Berkeley is a good enough filter.
Where did he say that? Given that he's working at UC Berkeley I would expect him to treat UC Berkeley students preferentially for reasons that aren't just about UC Berkeley being able to filter. It's natural that you can sign up for one of the classes he teaches at UC Berkeley by being a student of UC Berkeley. Being enrolled into MIT might be just as hard as being enrolled into UC Berkeley but it doesn't give you the same access to courses taught at UC Berkeley by it's faculty.
2philip_b2y [] and also
Okay, he does speak about using Berkeley as a filter but he doesn't speak about taking people as his student. It seems about helping people in UC Berkeley to connect with other people in UC Berkeley.

Methods, famously, includes the line "I am a descendant of the line of Bacon", tracing empiricism to either Roger (13th century) or Francis (16th century) (unclear which).

Though a cursory wikiing shows an 11th century figure providing precedents for empiricism! Alhazen or Ibn al-Haytham worked mostly optics apparently but had some meta-level writings about the scientific method itself. I found this shockingly excellent quote

The duty of the man who investigates the writings of scientists, if learning the truth is his goal, is to make himself an enemy of a

... (read more)

New discord server dedicated to multi-multi delegation research

DM me for invite if you're at all interested in multipolar scenarios, cooperative AI, ARCHES, social applications & governance, computational social choice, heterogeneous takeoff, etc.

(side note I'm also working on figuring out what unipolar worlds and/or homogeneous takeoff worlds imply for MMD research).

Questions and Problems - thoughts on chapter 4 of Craft of Doing Research

Last time we discussed the difference between information and a question or a problem, and I suggested that the novelty-satisfied mode of information presentation isn't as good as addressing actual questions or problems. In chapter 3 which I have not typed up thoughts about, A three step procedure is introduced

  1. Topic: "I am studying ..."
  2. Question: "... because I want to find out what/why/how ..."
  3. Significance: "... to help my reader understand ..." As we elaborate on the different k
... (read more)

preorders as the barest vocabulary for emergence

We can say "a monotonic map, is a phenomenon of as observed by ", then, emergence is simply the impreservation of joins.

Given preorders and , we say a map in "preserves" joins (which, recall, are least upper bounds) iff where by "" we mean .

Suppose is a measurement taken from a particle. We would like for our measurement system to be robust against emergence, which is literally operationalized by measuring one particle, measuring another, t... (read more)

Jotted down some notes about the law of mad science on the EA Forum. Looks like some pretty interesting open problems in the global priorities, xrisk strategy space.

Ambition, romance, kids

Two premises of mine are that I'm more ambitious than nearly everyone I meet in meatspace and normal distributions. This implies that in any relationship, I should expect to be the more ambitious one.

I do aspire to be a nagging voice increasing the ambitions of all my friends. I literally break the ice with acquaintances by asking "how's your master plan going?" because I try to create vibes like we're having coffee in the hallway of a supervillain conference, and I like to also ask "what harder project is your current project a war... (read more)

Positive and negative longtermism

I'm not aware of a literature or a dialogue on what I think is a very crucial divide in longtermism.

In this shortform, I'm going to take a polarity approach. I'm going to bring each pole to it's extreme, probably each beyond positions that are actually held, because I think median longtermism or the longtermism described in the Precipice is a kind of average of the two.

Negative longtermism is saying "let's not let some bad stuff happen", namely extinction. It wants to preserve. If nothing gets better for the poor or the an... (read more)

The audience models of research - thoughts on Craft of Doing Research chapter 2

Writers can't avoid creating some role for themselves and their readers, planned or not

Before considering the role you're creating for your reader, consider the role you're creating for yourself. Your broad options are the following

  1. I've found some new and interesting information - I have information for you
  2. I've found a solution to an important practical problem - I can help you fix a problem
  3. I've found an answer to an important question - I can help you understand somethi
... (read more)

Is there an EV monad? I'm inclined to think there is not, because EV(EV(X)) is a way simpler structure than a "flatmap" analogue.

Would there be a way of estimating how many people within the amazon organization are fanatical about same day delivery ratio against how many are "just working a job"? Does anyone have a guess? My guess is that an organization of that size with a lot of cash only needs about 50 true fanatics, the rest can be "mere employees". What do yall think?

I can't really think of any research bearing on this, and unclear how you'd measure it anyway. One way to go might be to note that there is a wide (and weird) variance between the efficiency of companies: market pressures are slack enough that two companies doing as far as can be told the exact same thing in the same geographic markets with the same inputs might be almost 100% different (I think was the range in the example of concrete manufacturing in one paper I read); a lot of that difference appears to be explainable by the quality of the management, and you can do randomized experiments in management coaching or intensity of management and see substantial changes in the efficiency of a company [] (Bloom - the other one - has a bunch of studies like this). Presumably you could try to extrapolate from the effects of individuals to company-wide effects, and define the goal of the 'fanatical' as something like 'maintaining top-10% industry-wide performance': if educating the CEO is worth X percentiles and hiring a good manager is worth 0.0Y percentiles and you have such and such a number of each, then multiply out to figure out what will bump you 40 percentiles from an imagined baseline of 50% to the 90% goal. Another argument might be a more Fermi estimate style argument from startups. A good startup CEO should be a fanatic about something, otherwise they probably aren't going to survive the job. So we can assume one fanatic at least. People generally talk about startups beginning to lose the special startup magic of agility, focus, and fanaticism at around Dunbar's number level of employees like 300, or even less (eg Amazon's two-pizza rule which is I guess 6 people?). In the 'worst' case that the founder has hired 0 fanatics, that implies 1 fanatic can ride herd over no more than ~300 people; in the 'best' case that he's hired dozens, then each fanatic can only cover for more like 2 or 3 non-fanatics. I'm
I'm not sure "fanatical" is well-defined enough to mean anything here. I doubt there are any who'd commit terrorist acts to further same-day delivery. There are probably quite a few who believe it's important to the business, and a big benefit for many customers. You're absolutely right that a lot of employees and contractors can be "mere employees", not particularly caring about long-term strategy, customer perception, or the like. That's kind of the nature of ALL organizations and group behaviors, including corporate, government, and social groupings. There's generally some amount of influencers/selectors/visionaries, some amount of strategists and implementers, and a large number of followers. Most organizations are multidimensional enough that the same people can play different roles on different topics as well.
I don't think it needs any true fanatics. It just needs incentives. This isn't to say there won't be fanatics anyway. There probably aren't many things that nobody can get fanatical about. This is even more true if they're given incentives to act fanatical about it.
Sure, but the incentive structure needs continual maintenance to keep it aligned with or pointing at the goal, which naturally leads to the questions of how many people are needed to keep the structure pointing at the goal, and what the motivation of those people will be.

We need a name for the following heuristic, I think, I think of it as one of those "tribal knowledge" things that gets passed on like an oral tradition without being citeable in the sense of being a part of a literature. If you come up with a name I'll certainly credit you in a top level post!

I heard it from Abram Demski at AISU'21.

Suppose you're either going to end up in world A or world B, and you're uncertain about which one it's going to be. Suppose you can pull lever which will be 100 valuable if you end up in world A, or you can pull lever whi... (read more)

Why are you specifying 100 or 0 value, and using fuzzy language like "acceptably small" for disvalue? Is this based on "value" and "disvalue" being different dimensions, and thus incomparable? Wouldn't you just include both in your prediction, and run it through your (best guess of) utility function and pick highest expectation, weighted by your probability estimate of which universe you'll find yourself in?
100 and 0 in this context make sense. Or at least in my initial reading: arbitrarily-chosen values that are in a decent range to work quickly with (akin to why people often work in percentages instead of 0..1) It is - I'm going to say "often", although I am aware this is suboptimal phrasing - often the case that you are confident in the sign of an outcome but not the magnitude of the outcome. As such, you can often end up with discontinuities at zero. Dropping the entire probability distribution of outcomes through your utility function doesn't even necessarily have a closed-form result. In a universe where computation itself is a cost, finding a cheaper heuristic (and working through if said heuristic has any particular basis or problems) can be valuable. The heuristic in the grandparent comment is just what happens if you are simultaneously very confident in the sign of positive results, and have very little confidence in the magnitude of negative results.
It is often the case that you are confident in the sign of an outcome but not the magnitude of the outcome. This heuristic is what happens if you are simultaneously very confident in the sign of positive results, and have very little confidence in the magnitude of negative results.
I'm not sure I understand. If the lever is +100 in world A and -90 in world B, it seems like a good bet if you don't know which world you're in. Or is that what you mean by "acceptably small amount of disvalue"?
Obviously there are considerations downstream of articulating this, one is that when P(A)>P(B) but V(LA|A)<V(LB|B) so it's reasonable to hedge on ending up in world B even though it's not strictly more probable than ending up in world A.

critiques and complaints

I think one of the most crucial meta skills i've developed is honing my sense of who's criticizing me vs. who's complaining.

A criticism is actionable, implicitly often it's from someone who wants you to win. A complaint is when you can't figure out how you'd actionably fix something or improve based on what you're being told.

This simple binary story is problematic. It can empower you to ignore criticism you don't like by providing a set of excuses, if you're not careful. Sometimes it's operationally impossible to parse out a critic... (read more)

hmu for a haskell job in decentralized finance. Super fun zero knowledge proof stuff, great earning to give opportunity.

Are shelling points the occam's razor of mechanism design?

In game theory, a focal point (or Schelling point) is a solution that people tend to choose by default in the absence of communication. (wikipedia)

Intuitively I think simplicity is a good explanation for a solution being converged upon.

Does anyone have any crisp examples that violate the schelling point - occam's razor correspondence?

Anchoring effect is enough for a Schelling point, it doesn't have to be simple solution. For instance a new nation that wants to move away from dictatorship is automatically going to build a democracy with multiple independent arms (legislature, judiciary, executive), a constitution, periodic elections of representatives, etc. They could choose to try a direct democracy or change the term from 5 years to 1 year, they could choose to have public elections for the judiciary too, or any other deviation from how democracies usually run, but they won't. Fear of the unknown + no creativity or motivation will be sufficient from them to copy existing countries' democratic structure.

Disvalue via interpersonal expected value and probability

My deontologist friend just told me that treating people like investments is no way to live. The benefits of living by that take are that your commitments are more binding, you actually do factor out uncertainty, because when you treat people like investments you always think "well someday I'll no longer be creating value for this person and they'll drop me from their life". It's hard to make long term plans, living like that.

I've kept friends around out of loyalty to what we shared 5-10 years ago w... (read more)

One thing to be careful about in such decisions - you don't know your own utility function very precisely, and your modeling of both future interactions and your value from such are EXTREMELY lossy. The best argument for deontological approaches is that you're running on very corrupt hardware, and rules that have evolved and been tested over a long period of time are far more trustworthy than your ad-hoc analysis which privileges obvious visible artifacts over more subtle (but often more important) considerations.
Imo choosing to disconnect from people who are no longer providing any value to you is just a healthy thing to do, even a deontologist should agree with that.

I may refine this into a formal bounty at some point.

I'm curious if censorship would actually work in the context of blocking deployment of superpowerful AI systems. Sometimes people will mention "matrix multiplication" as a sort of goofy edge case, which isn't very plausible, but that doesn't mean there couldn't be actual political pressure to censor it. A more plausible example would be attention. Say the government threatens soft power against arxiv if they don't pull attention is all you need, or threatens soft power against harvard if their linguistic... (read more)

You'll need a govt body full of people who are aligned in their thinking, no one should defect. Also Yudkowsky's response to this would prolly be that it isn't enough to censor the first time the idea is created, someone else will just discover another (or the same) path to AGI independently. See pivotal act [].

any literature on estimates of social impact of businesses divided by their valuations?

the idea that dollars are a proxy for social impact is neat, but leaves a lot of room for goodhart and I think it's plausible that they diverge entirely in cases. It would be useful to know, if possible to know, what's going on here.

1Josh Jacobson1y
there's paid tools that estimate this, probably poorly
thinking about this comment []

Why have I heard about Tyson investing into lab grown, but I haven't heard about big oil investing in renewable?

Tyson's basic insight here is not to identify as "an animal agriculture company". Instead, they identify as "a feeding people company". (Which happens to align with doing the right thing, conveniently!)

It seems like big oil is making a tremendous mistake here. Do you think oil execs go around saying "we're an oil company"? When they could instead be going around saying "we're a powering stuff" company. Being a powering stuff company means you hav... (read more)

Yes, this is more about you not hearing about it. SHELL HAS A BIGGER CLEAN ENERGY PLAN THAN YOU THINK — CLEANTECHNICA INTERVIEW [HTTPS://CLEANTECHNICA.COM/2020/05/01/SHELL-HAS-A-BIGGER-CLEAN-ENERGY-PLAN-THAN-YOU-THINK-CLEANTECHNICA-INTERVIEW/] BP BETS FUTURE ON GREEN ENERGY, BUT INVESTORS REMAIN WARY [HTTPS://WWW.WSJ.COM/ARTICLES/BP-BETS-FUTURE-ON-GREEN-ENERGY-BUT-INVESTORS-REMAIN-WARY-11601402304] It seems that Tyson invested 150 million [] into a fund for new food solutions. In contrast to that Exxon invested 600 million []in algae biofuels back in 2009 and more afterward.
2Yoav Ravid1y
I do vaguely remember hearing of big oil doing that, though perhaps not as much as meat producers do with lab grown meat, try looking into it.
1. Might be a little bit harder in that industry. 2. Are they in charge (of that)? Who chose them?
you're most likely right about it being harder in the industry! I don't think they need permission or an external mandate to do the right thing!
The main problem is that prior investment into the oil method of powering stuff doesn't translate into having a comparative advantage in a renewable way of powering stuff. They want a return on their existing massive investments. While this looks superficially like a sunk cost fallacy, it isn't. If a comparatively small investment (mere billions) can ensure continued returns on their trillions of sunk capital for another decade, it's worth it to them. Investment into renewable powering stuff would require substantially different skill sets in employees, in very different locations, and highly non-overlapping investment. At best, such an endeavour would constitute a wholly owned subsidiary that grows while the rest of the company withers. At worst, a parasite that hastens the demise of the parent while eventually failing in the face of competition anyway.

I've had a background assumption in my interpretation of and beliefs about reward functions for as long as I can remember (i.e. since first reading the sequences), that I suddenly realized I don't believe is written down. Over the last two years I've gained experience writing coq sufficient to inspire a convenient way of framing it.

Computational vs axiomatic reward functions

Computational vs axiomatic in proof engineering

A proof engineer calls a proposition computational if it's proof can be broken down into parts.

For example, a + (b + c) = (a + b) + c i... (read more)

I should be more careful not to imply I think that we have solid specimens of computational reward functions; more that I think it's a theoretically important region of the space of possible minds, and might factor in idealizations of agency

capabilities-prone research.

I come to you with a dollar I want to spend on AI. You can allocate p pennies to go to capabilities and 100-p pennies to go to alignment, but only if you know of a project that realizes that allocation. For example, we might think that GAN research sets p = 98 (providing 2 cents to alignment) while interpretability research sets p = 10 (providing 90 cents to alignment).

Is this remotely useful? This is a really rough model (you might think it's more of a venn diagram and that this model doesn't provide a way of reasoning about t... (read more)

Question your argument as your readers will - thoughts on chapter 10 of Craft of Research

Three predictable disagreements are

  • There are causes in addition to the one you claim
  • What about these counterexamples?
  • I don't define X as you do, to me X means...

There are roughly two kinds of queries readers will have about your argument

  1. intrinsic soundness - "challenging the clarity of a claim, relevance of reasons, or quality of evidence"
  2. extrinsic soundness - "different ways of framing the problem, evidence you've overlooked, or what others have written on t
... (read more)

there's a gap in my inside view of the problem, part of me thinks that capabilities progress such as out-of-distribution robustness or the 4 tenets described in open problems in cooperative ai is necessary for AI to be transformative, i.e. a prereq of TAI, and another part of me that thinks AI will be xrisky and unstable if it progresses along other aspects but not along the axis of those capabilities.

There's a geometry here of transformative / not transformative cross product with dangerous not dangerous.

To have an inside view I must be able to adequately navigate between the quadrants with respect to outcomes, interventions, etc.

If something can learn fast enough, then it's out-of-distribution performance won't matter as much. (OOD performance will still matter -but it'll have less to learn where it's good, and more to learn where it's not.*) *Although generalization ability seems like the reason learning matters. So I see why it seems necessary for 'transformation'.

testing latex in spoiler tag

Testing code block in spoiler tag

::: latex Ax+1:={} :::

missed opportunities to build a predictive track record and trump

I was reminiscing about my prediction market failures, the clearest "almost won a lot of mana dollars" (if manifold markets had existed back then) was this executive order. The campaign speeches made it fairly obvious, and I'm still salty about a few idiots telling me "stop being hysterical" when I accused him of being what he's writing on the tin that he is pre inauguration even though I overall reminisce that being a time when my epistemics were way worse than they are now.

However, there d... (read more)

New to LessWrong?