# All of Quinn's Comments + Replies

johnswentworth's Shortform

You might check out Donald Braben's view, it says "transformative research" (i.e. fundamental results that create new fields and industries) is critical for the survival of civilization. He does not worry that transformative results might end civilization.

Quinn's Shortform

# Question your argument as your readers will - thoughts on chapter 10 of Craft of Research

Three predictable disagreements are

• There are causes in addition to the one you claim
• I don't define X as you do, to me X means...

1. intrinsic soundness - "challenging the clarity of a claim, relevance of reasons, or quality of evidence"
2. extrinsic soundness - "different ways of framing the problem, evidence you've overlooked, or what others have written on t
Quinn's Shortform

I asked a friend whether I should TA for a codeschool called ${{codeschool}}. You shouldn't hang around${{codeschool}}. People at ${{codeschool}} are not pursuing excellence. A hidden claim there that I would soak up the pursuit of non-excellence by proximity or osmosis isn't what's interesting (though I could see that turning out either way). What's interesting is the value of non-excellence, which I'll call adequacy.${{codeschool}} in this case is effective and impactful at putting butts in seats at companies, and is thereby re... (read more)

2Viliam23dSeems to me that on the market there are very few jobs for the SICP types. The more meta something is, the less of that is needed. If you can design an interactive website, there are thousands of job opportunities for you, because thousands of companies want an interactive website, and somehow they are willing to pay for reinventing the wheel. If you can design a new programming language and write a compiler for it... well, it seems that world already has too many different programming languages, but sure there is a place for maybe a dozen more. The probability of success is very small even if you are a genius. The best opportunity for developers who think too meta is probably to design a new library for an already popular programming language, and hope it becomes popular. The question is how exactly you plan to get paid for that. Probably another problem is that it requires intelligence to recognize intelligence, and it requires expertise to recognize expertise. The SICP type developer seems to most potential employers and most potential colleagues as... just another developer. The company does not see individual output, only team output; it does not matter that your part of code does not contain bugs, if the project as a whole does. You cannot use solutions that are too abstract for your colleagues, or for your managers. Companies value replaceability, because it is less fragile and helps to keep developer salaries lower than they might be otherwise. (In theory, you could have a team full of SICP type developers, which would allow them to work smarter, and yet the company would feel safe. In practice, companies can't recognize this type and don't appreciate it, so this is not going to happen.) Again, probably the best position for a SICP type developer in a company would be to develop some library that the rest of the company would use. That is, a subproject of a limited size that the developer can do alone, so they are not limited in the techniques they us
Quinn's Shortform

# thoughts on chapter 9 of Craft of Research

Getting the easy things right shows respect for your readers and is the best training for dealing with the hard things.

If they don't believe the evidence, they'll reject the reasons and, with them, your claim.

We saw previously that claims ought to be supported with reasons, and reasons ought to be based on evidence. Now we will look closer at reasons and evidence.

Quinn's Shortform

# Claims - thoughts on chapter eight of Craft of Research

Broadly, the two kinds of claims are conceptual and practical.

Conceptual claims ask readers not to ask, but to understand. The flavors of conceptual claim are as follows:

• Claims of fact or existence
• Claims of definition and classification
• Claims of cause and consequence
• Claims of evaluation or appraisal

There's essentially one flavor of practical claim

• Claims of action or policy.

If you read between the lines, you might notice that a kind of claim of fact or cause/consequence is that a policy work... (read more)

2Viliam25dThis may be context-dependent. Different countries probably have different cultural norms. Norms may differ for higher-status and lower-status speakers. Humble speech may impress some people, but others may perceive it as a sign of weakness. Also, is your audience fellow scientists or are you writing a popular science book? (More hedging for the former, less hedging for the latter.)
Quinn's Shortform

# Good arguments - notes on Craft of Research chapter 7

Arguments take place in 5 parts.

1. Claim: What do you want me to believe?
2. Reasons: Why should I agree?
3. Evidence: How do you know? Can you back it up?
4. Acknowledgment and Response: But what about ... ?
5. Warrant: How does that follow?

This can be modeled as a conversation with readers, where the reader prompts the writer to taking the next step on the list.

Claim ought to be supported with reasons. Reasons ought to be based on evidence. Arguments are recursive: a part of an argument is an acknowledgment of... (read more)

Quinn's Shortform

# Sources - notes on Craft of Research chapters 5 and 6

## Primary, secondary, and tertiary sources

Primary sources provide you with the "raw data" or evidence you will use to develop, test, and ultimately justify your hypothesis or claim. Secondary sources are books, articles, or reports that are based on primary sources and are intended for scholarly or professional audiences. Tertiary sources are books and articles that synthesize and report on secondary sources for general readers, such as textbooks, articles in encyclopedias, and articles in mass-circulat

Quinn's Shortform

# Questions and Problems - thoughts on chapter 4 of Craft of Doing Research

Last time we discussed the difference between information and a question or a problem, and I suggested that the novelty-satisfied mode of information presentation isn't as good as addressing actual questions or problems. In chapter 3 which I have not typed up thoughts about, A three step procedure is introduced

1. Topic: "I am studying ..."
2. Question: "... because I want to find out what/why/how ..."
3. Significance: "... to help my reader understand ..." As we elaborate on the different k
Quinn's Shortform

# The audience models of research - thoughts on Craft of Doing Research chapter 2

Writers can't avoid creating some role for themselves and their readers, planned or not

1. I've found some new and interesting information - I have information for you
2. I've found a solution to an important practical problem - I can help you fix a problem
2Michaël Trazzi1modone [https://www.stitcher.com/show/630264]! should be live in a few hours
[timeboxed exercise] write me your model of AI human-existential safety and the alignment problems in 15 minutes

Given that systems of software which learn can eventually bring about 'transformative' impact (defined as 'impact comparable to the industrial revolution'), the most important thing to work on is AI. Given that the open problems in learning software between now and its transformativity can be solved in a multitude of ways, some of those solutions will be more or less beneficial, less or more dangerous, meaning there's a lever that altruistic researchers can use to steer outcomes in these open problems. Given the difficulty of social dilemmas and coordinati... (read more)

Quinn's Shortform

there's a gap in my inside view of the problem, part of me thinks that capabilities progress such as out-of-distribution robustness or the 4 tenets described in open problems in cooperative ai is necessary for AI to be transformative, i.e. a prereq of TAI, and another part of me that thinks AI will be xrisky and unstable if it progresses along other aspects but not along the axis of those capabilities.

There's a geometry here of transformative / not transformative cross product with dangerous not dangerous.

To have an inside view I must be able to adequately navigate between the quadrants with respect to outcomes, interventions, etc.

2Pattern1moIf something can learn fast enough, then it's out-of-distribution performance won't matter as much. (OOD performance will still matter -but it'll have less to learn where it's good, and more to learn where it's not.*) *Although generalization ability seems like the reason learning matters. So I see why it seems necessary for 'transformation'.
We need a career path for invention

You might like Scientific Freedom by Donald Braben. It's a whole book about the problem of developing incentives for basic research.

Quinn's Shortform

# notes (from a very jr researcher) on alignment training pipeline

Training for alignment research is one part competence (at math, cs, philosophy) and another part having an inside view / gears-level model of the actual problem. Competence can be outsourced to universities and independent study, but inside view / gears-level model of the actual problem requires community support.

A background assumption I'm working with is that training as a longtermist is not always synchronized with legible-to-academia training. It might be the case that jr researchers oug... (read more)

2ChristianKl1moI don't think Critch's saying that the best way to get his attention is through cold emails backed up by credentials. The whole post is about him not using that as a filter to decide who's worth his time but that people should create good technical writing to get attention.
Announcing the Technical AI Safety Podcast

Thanks for reaching out! Alex had passed onto me the note about transcripts, I hope to get to it (including the backlog of already released episodes) in the next few months.

Averting suffering with sentience throttlers (proposal)

Right, I feel like there's a tradeoff between interestingness of consciousness theory and the viability of computational predicates. IIT gives you a nice computer program, but isn't very interesting.

Could degoogling be a practice run for something more important?

I think the litmus test for the value of reducing dependency on a given product/technology is whether we think it's empowering or enfeebling. Consider arithmetic calculators: is it empowering to delegate boring stuff to subroutines freeing up your mind to do harder stuff, or is it enfeebling because it reduces incentive to learn to do mental arithmetic well? Dependence can be a problem in either case.

Each product needs to be assessed individually.

Open and Welcome Thread - April 2021

I'm trying to decide if i'm going to write up a thought about longtermism I had.

I think there are two schools of thought-- that the graph of a value function over time is continuous or discontinuous. The continuous school of thought suggests that you get near term evidence about long term consequences, and the discontinuous school of thought does not interpret local perturbation in this way at all.

I'm sure this is covered in one of the many posts about longtermism, and the language of continuous functions could either make it clearer or less clear depending on the audience.

1eigen2moI don't think there's enough written about long-termism. You have a reader here if you ever decide to write something. I wonder as to where between those two school of thoughts you fall.
Takeaways from the Intelligence Rising RPG

I can't post a complete ruleset, but I can add some insight-- each party had "stats" representing hard power, soft power, budget, that sort of thing. Each turn you could spend "talent" stats on research arbitrarily, and you could take two "actions" which were GM-mediated expenditures of things like soft power, budget, etc. The game board was a list of papers and products that could be unlocked, unlocking papers released new products onto the board

Reading recommendations on social technology: looking for the third way between technocracy and populism

isn't increasing the competence of the voter akin to increasing the competence of the official, by proxy? I'm pattern matching this to yet another push-pull compromise between the ends of the spectrum, with a strong lean toward technocracy's side.

I'm assuming I'll have to read Brennan for his response to the criticism that it was tried in u.s. and made a lot of people very upset / is widely regarded as a bad move.

I agree with Gerald Monroe about the overall implementation problems even if you assume it wouldn't just be a proxy for race or class war (which ... (read more)

0vernamcipher4moImplementation problems are definitely a problem with Brennan's Knowledge Test To Vote idea and consist of two parts: (1) getting the present voters to agree to it (2) setting a test that is discriminatory in the right rather than the wrong ways. One would hope a good answer to (2) would help with (1), though convincing people to give up the vote would be very hard. I have been thinking a fair bit lately about the content of a Voting Test. Presumably one would want tests of knowledge that are proxies for being what Brennan calls a Vulcan - an informed Non-partisan voter who considers things like evidence - rather than a Hooligan - informed partisan - or Hobbit - uninformed and nonpartisan. Brennan's idea to test for basic knowledge about government is a good start - how does a bill become law, how do the different branches of government work, how much does your country spend on foreign aid as a percentage of government expenditures (the latter being something surveyed voters consistently and overwhelmingly get wrong). I would add to such a test sections for basic probability, statistics, and economics as these are vital for understanding public policy issues. Anyone who thinks the difference between 2% annual GDP growth and 3% annual GDP growth is 1% has next to nothing to contribute to public discourse.
Scott and Rohin doublecrux on AI with human models

should i be subscribed to a particular youtube channel where these things get posted?

4Ben Pace4moWe occasionally post them to the LessWrong YouTube channel: https://www.youtube.com/channel/UCt7yNB7lIW89N0hvISKJMpA [https://www.youtube.com/channel/UCt7yNB7lIW89N0hvISKJMpA] But mostly the best place to check is the event post (like this one), where it will eventually be updated.
Anki decks by LW users

Quick Bayes Table, by alexvermeer. A simple deck of cards for internalizing conversions between percent, odds, and decibels of evidence.

2Pablo3moThanks, but this post is no longer updated and the link is not broken on my website. (If you think that's confusing, despite the notice at the top, I may consider replacing its contents with just a link, though retaining the content may make it more discoverable.)
2TurnTrout4moThe spoiler seems to be empty?
Lessons I've Learned from Self-Teaching

Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on.

This is hard to instrumentalize regarding difficulty. I find that the hardest exercises are likeliest to be skipped (after struggling with them for an hour or two), but it doesn't follow that I can expect the easier ones (which I happened to have completed) to lie in that key 20%.

2TurnTrout5moI like to randomly sample X% of the exercises, and read the rest; this lets me know later whether or not I missed something important. Simple rules like "do every fifth exercise" should suffice, with the rule tailored to the number of exercises in the book.
Quinn's Shortform

::: latex :::

Quinn's Shortform

:::hm? x :: Bool -> Int -> String :::

1Quinn5mo::: latex Ax+1:={} :::
Quinn's Shortform

testing latex in spoiler tag

Testing code block in spoiler tag

Infodemics: with Jeremy Blackburn and Aviv Ovadya

7p on thursday the 14th for New York, 4p in San Fransisco

Announcing the Technical AI Safety Podcast

When I submitted to pocketcasts it said we were already on it :) https://pca.st/9froevor

Have general decomposers been formalized?

Thank you Abram. Yes, factored cognition is more what I had in mind. However, I think it's possible to speak of decomposition generally enough to say that PCA/SVD is a decomposer, albeit an incredibly parochial one that's not very useful to factored cognition.

Like, my read of IDA is that the distillation step is proposing a class of algorithms, and we may find that SVD was a member of that class all along.

How ought I spend time?

I'll check out Lynette's post.

I'd like to take a shot at technical AI alignment

How ought I spend time?

What granularity of time are you talking about? When you "never maintain 1 and 2 at the same time", is that any given minute, or any given decade?

I would say every couple months is an opportunity to either pivot or continue.

Have general decomposers been formalized?

Sorry, I think I might have a superficial understanding of encoders and embeddings. Would you be able to try pointing out for me how decomposition is performed in that case (or point me toward a favorite reading on the subject)? When I think of feeding a sentence into an encoder, I can think of multiple ways in which some compositional structure might be inferred.

I'm drawing up a proof of concept with seq2seq learners right now, but my hypothesis is that they will be inadequate decomposers suitable only for benchmarking a baseline.

3George1yI was asking why because I wanted to understand what you mean by "decomposition". Defines many things. Usually the goal is feature extraction (think Bert) or reducing the size of a representation (think autoencoders or simpler , PCA) You need to narrow down your definition, I think, to get a meaningful answers.
The Politics of Age (the Young vs. the Old)

SITG-suffrage Sorry, by this point OP and I had established "right to vote weighted by stake" as a concept, using the words "skin-in-the-game", so SITG was an acronym for skin-in-the-game, and suffrage referred to right to vote.

Parents are different from any other group in my comment because I was referencing Richard Kennaway's question "Does having children whose future you care about also count as skin in the game?"

The Unexpected Philosophical Depths of the Clicker Game Universal Paperclips

A year or two before the paperclip version came out I played a lot of AdVenture Capitalist (and it's sequel, wait for it, AdVenture Communist), was wondering to myself whether reinforcement learning researchers would find it interesting, and wondering if deep mind would start training up agents to compete in AdVenture Capitalist tournaments.

The Politics of Age (the Young vs. the Old)

Does having children whose future you care about also count as skin in the game?

Unclear. There's a lot to unpack, because we don't know the 1. narcissism or 2. epistemic competence distributions across parents. I.e., we can't expect that what parents' say are in their kids' interests actually share their kids' interests (either through willful misdirection or through earnest mistakes).

Or you can say that your skin-in-the-game factor is proprotional to how much you've already invested in the status quo. If you've spent 50 years working towards a goal i

2Dagon2yor across non-parents, or old people, or teenagers, or any other group. If we think we CAN measure them well, we should just measure them and set voting standards for individuals, not age-based demographic groups (though I'd be fine with a combo: everyone can vote between 25 and 65, and anyone who passes the competence/non-narcissism/whatever threshold can vote regardless of age). Not familiar with the term, and Google doesn't show anything that looks relevant on the first few pages of "SITG" suffrage. I assume this is the theory "landholders are the only ones with standing to care about the land, and they happen to be the rich and powerful" idea. If you don't mean to guilt-by-association an argument, then please don't do so. I dispute the assumption that 70-year olds only care about the same things that the previous cohort did, and not about the things they cared about as 60-year-olds. That caricature is at least as bad as saying 16-year-olds care about the same things that all 16-year-olds have cared about forever (sex/freedom/unearned respect/bad music). I'd argue there's more truth in the latter, but not enough truth to make a valid argument. I'd also like to point out that dotards select themselves out of voting by not having spare energy to participate. The young and stupid/naive have no such selection mechanism.
The Politics of Age (the Young vs. the Old)

A skin-in-the-game vote multiplier based on age might look like mean lifespan - your age. That's the logical consequence of saying that people who have to put up with outcomes longer ought to weigh higher in shaping them. It should floor out at around 1 at the upper limit, and the lower limit should come from enforceability of anti-fraud measures (i.e. effectiveness at stopping parents from using kids who can't walk yet for extra votes) instead of from anyone's intuitions about when kids can think for themselves.

If some experts got together and said that b

7Richard_Kennaway2yDoes having children whose future you care about also count as skin in the game?
8Martin Sustrik2yThere are two opposing ways to think about it. You can either, as you do, say that your skin-in-the-game is proportional to the amount of time you have in front of you. From that perspective it seems fair that children should have biggest say in shaping long-term policies. Or you can say that your skin-in-the-game factor is proprotional to how much you've already invested in the status quo. If you've spent 50 years working towards a goal it seems unfair that a 16-year old know-nothing should be able, on a whim, to throw all of that away.
Do the best ideas float to the top?

IMO, this is what I briefly suggested by linking to Scott's Against Murderism with the words "misleading compression", i.e., I think describing a policy as murderistic and optimizing for stories are each instances of misleading compression.

If it’s only stories which matter, yet you split your efforts between stories and reality, then you will likely be outcompeted by someone who spent all of their resources on crafting good stories.

This is 100% what I find alarming about misinformation (both the malicious kind and the emergent/inadequate kind), and I don't know a reason why alignment via debate would be resilient.

Do the best ideas float to the top?

Sorry. The point was NAT, density_{1,2,3} was devised scaffolding for the MVB (minimum viable blogpost). I imagine that NAT has already been discovered, discussed, problematized etc. somewhere but I couldn't find it. I have a background assumption that attention economists are competent and well-intentioned people, so I trust that they have the situation under control.

Do the best ideas float to the top?

Likewise, what level do you want a NAT to be implemented at? Personal behavior? Structure of group blog sites? Social norms?

• personal behavior: probably not viable without a dystopian regime of microchips embedded into brains.

• structure of group blog sites: maybe-- these things have been suggested and tried, i.e. I can't tell you how many times I've seen a reddit comment lamenting the incentives of their upvote system.

• weirdly, I found out about the Brave browser last week (weird because it's apparently been around for a wh

Functional silence: communication that minimizes change of receiver's beliefs

Not much to add, but: Yes, you nailed it, I see it in the world all the time.

Do the best ideas float to the top?

This one had slipped by me, so thanks for pointing me to it. It'll take me at least a week to read and digest. I'll add a comment here (eventually) if I have anything to say.

Do the best ideas float to the top?

I don't know a lot about evolution, but I suspect any benefits of building on memetics work directly would fall under the umbrella of "what about when we're tipping the scale in favor of some ingroup?". I defined density_3 as a placeholder for this along with all maximization related issues, and then said "we'll ignore this for now and focus on more basic foundations". I don't know if I'll return to it, but if I do, it'll take me a really really long time.

1Matt Goldenberg2yThe thing I was trying to point at is that memetics IS the basic foundations. All three of the items you mentioned are a side effect of survival and replication characteristics, not something that underlie them. It may be that the work you're trying to do here has already been done.
What makes people intellectually active?

I'm not trying to hold it constant, I'm just trying to understand a relatively low standard, because that's the part I feel confused about. It seems relatively much easier to look at bad intellectual output and say how it could have been better, think about the thought processes involved, etc. Much harder to say what goes into producing output at all vs not doing so.

I think I understand the distinction, and I think if it was as simple as "people undershoot their actual capacities in favor of humility / don't want to risk wasting anybody's time" everyone would have adjusted social norms to remedy it by now.

thanks

What makes people intellectually active?

I was thinking in a very different direction upon reading "a lot of people also find that writing down your ideas, causes you to have even more ideas." I know what you mean in the context of a reinforcement system, but I think it misses the more pressing phenomena, at least in my experience of uncertainty whether i'm inventing or indulging, of working on ideas.

The "even more ideas" part sounds to me like a sort of (combinatorial) explosion, when my stroke of inspiration is much more problematic, much less elegant than I thought. Sometimes this also means m