Shortform Content [Beta]

rohinmshah's Shortform

I often have the experience of being in the middle of a discussion and wanting to reference some simple but important idea / point, but there doesn't exist any such thing. Often my reaction is "if only there was time to write an LW post that I can then link to in the future". So far I've just been letting these ideas be forgotten, because it would be Yet Another Thing To Keep Track Of. I'm now going to experiment with making subcomments here simply collecting the ideas; perhaps other people will write posts about them at some point, if they're even understandable.

Showing 3 of 28 replies (Click to show all)

Let's say you're trying to develop some novel true knowledge about some domain. For example, maybe you want to figure out what the effect of a maximum wage law would be, or whether AI takeoff will be continuous or discontinuous. How likely is it that your answer to the question is actually true?

(I'm assuming here that you can't defer to other people on this claim; nobody else in the world has tried to seriously tackle the question, though they may have tackled somewhat related things, or developed more basic knowledge in the domain that you can leverage.)

F... (read more)

4rohinmshah2moSometimes people say "look at these past accidents; in these cases there were giant bureaucracies that didn't care about safety at all, therefore we should be pessimistic about about AI safety". I think this is backwards, and that you should actually conclude the reverse: this is evidence that problems tend to be easy, and therefore we should be optimistic about AI safety. This is not just one man's modus ponens [] -- the key issue is the selection effect. It's easiest to see with a Bayesian treatment. Let's say we start completely uncertain about what fraction of people will care about problems, i.e. uniform distribution over [0, 100]%. In what worlds do I expect to see accidents where giant bureaucracies don't care about safety? Almost all of them -- even if 90% of people care about safety, there will still be some cases where people didn't care and accidents happened; and of course we'd hear about them if so (and not hear about the cases where accidents didn't happen). You can get a strong update against 99.9999% and higher, but by the time you're at 90% the update seems pretty weak. Given how much selection there is, I think even the update against 99% is relatively weak. So really you just don't learn much about how careful people will be by looking at our accident track record (unless you can also quantify the denominator of how many "potential accidents" there could have been). However, it feels pretty notable to me that the vast majority of accidents I hear about in detail are ones where it seems like there were a bunch of obvious mistakes and the accidents would have been prevented had there been a decision-maker who cared (enough) about safety. And unlike the previous paragraph, I do expect to hear about accidents that we couldn't have prevented, so I don't have to worry about selection bias. So it seems like I should conclude that usually problems are pretty easy, and "all we have to do" is make sure people care. (One counter
4rohinmshah2moYou've heard of crucial considerations, but have you heard of red herring considerations? These are considerations that intuitively sound like they could matter a whole lot, but actually no matter how the consideration turns out it doesn't affect anything decision-relevant. To solve a problem quickly, it's important to identify red herring considerations before wasting a bunch of time on them. Sometimes you can even start outlining solutions that turn a bunch of seemingly-crucial considerations into red herring considerations. For example, it might seem like "what is the right system of ethics" is a crucial consideration for AI alignment (after all, you need to know ethics to write down a utility function), but once you decide to instead aim to design algorithms that allow you to build AI systems for any task you have in mind, that turns into a red herring consideration. Here's an example [] where I argue that, for a specific question, anthropics is a red herring consideration (thus avoiding the question of whether to use SSA or SIA). Alternate names: sham considerations? insignificant considerations?
Raemon's Shortform

At any given time, is there anything especially wrong about using citation count (weighted by the weightings of other paper's citation count) as a rough proxy for "what are the most important papers, and/or best authors, weighted?"

My sense is the thing that's bad about this is that it creates an easy goodhart metric. I can imagine worlds where it's already so thoroughly goodharted that it doesn't signal anything anymore. If that's the case, can you get around that by grounding it out in some number of trusted authors, and purging obviously fraudulent autho... (read more)

4jimrandomh1dIt depends what you mean by "rough proxy", and whether you're applying it to scientific papers (where Goodhart has been out in force for decades, so a one-time check is off the table) or to LessWrong posts (where citation-count has never been something people cared about). Most things have zero citations, and this is indeed a negative quality signal. But after you get to stuff that's cited at all, citation count is mainly determined by the type and SEO of a paper, rather than its quality. Eg this paper [] . Citations also don't distinguish building upon something from criticizing it. That's much worse in the Goodhart arena than the one-time arena, but still pretty bad in the one-shot case.

Nod. "positive vs disagreement citation" is an important angle I wasn't thinking about.

1Zac Hatfield Dodds2dImportant for what? Best for what? In a given (sub)field, the highest-cited papers tend to be those which introduced or substantially improved on a key idea/result/concept; so they're important in that sense. If you're looking for the best introduction though that will often be a textbook, and there might be important caveats or limitations in a later and less-cited paper. I've also had a problem where a few highly cited papers propose $approach, many papers apply or puport to extend it, and then eventually someone does a well-powered study checking whether $approach actually works. Either way that's an important paper, but they tend to be under-cited either because either the results are "obvious" (and usually a small effect) or the field of $approach studies shrinks considerably. It's an extremely goodhartable metric but perhaps the best we have for papers; for authors I tend to ask "does this person have good taste in problems (important+tractable), and are their methods appropriate to the task?".
Jimrandomh's Shortform

On October 26, 2020, I submitted a security vulnerability report to the Facebook bug bounty program. The submission was rejected as a duplicate. As of today (April 14), it is still not fixed. I just resubmitted, since it seems to have fallen through the cracks or something. However, I consider all my responsible disclosure responsibilities to be discharged.

Once an Oculus Quest or Oculus Quest 2 is logged in to a Facebook account, its login can't be revoked. There is login-token revocation UI in Facebook's Settings>Security and Login menu, but changing t... (read more)

1BossSleepy2dIs your logic that releasing this heinous volun into the public is more likely to pressure FB to do something about this? Because if so, I'm not sure that LW is a forum with enough public spotlight to generate pressure. OTOH, I imagine some percentage of readers here aren't well-aligned but are looking for informational edge, in which case it's possible this does more harm than good? I'm not super-confident in this model -- eg, it also seems entirely possible to me that lots of FB security engineers read the site and one or more will be shouting ZOMG! any moment over this..

I'm posting here (cross-posted with my FB wall and Twitter) mostly to vent about it, and to warn people that sharing VR headsets has infosec implications they may not have been aware of. I don't think this comment will have much effect on Facebook's actions.

niplav's Shortform

Life is quined matter.

4Spiracular5dDNA is a quine, when processed by DNA Replicase. Although with a sufficiently complex (or specific) compiler or processor, any arbitrary stream of information can be made into a quine. You can embed information in either of instructions/substrate, or compiler/processor. Quines usually seem limited to describing instructions/substrate, but that's not the only place information can be drawn from. I've kinda come to think of it as 2-term operation (ex: "this bit of quine code is a quine wrt python"). (More physical quines: Ink scribbles on a sheet of paper are a quine wrt a copy-machine. At about the far-end of "replicative complexity gets stored in the processor," you have the "activated button" of a button-making machine (which activates with a button, and outputs buttons in an active state). I think the "activated button" here is either a quine, or almost a quine.) The cool thing about life is that it it is both a quine, and its own processor. (And also, sorta its own power source.) I find it simpler to call systems like this "living" (at least while active/functional), since they're meaningfully more than just quines. Viruses are definitely quines, though. Viruses and plasmids are non-living quines that compile and run on living biological systems.
1niplav3dIsn't life then a quine running on physics itself as a substrate? I hadn't considered thinking of quines as two-place, but that's obvious in retrospect [].

It's sorta non-obvious. I kinda poked at this for hours, at some point? It took a while for me to settle on a model I liked for this.

Here's the full notes for what I came up with.

Physics: Feels close. Hm... biological life as a self-compiler on a physics substrate?

DNA or gametes seem really close to a "quine" for this: plug it into the right part of an active compiler, and it outputs many instances of its own code + a customized compiler. Although it crashes/gets rejected if the compiler is too different (ex: plant & animal have different regulatory m... (read more)

Spiracular's Shortform Feed

Live Parsers and Quines

An unusual mathematical-leaning definition of a living thing.

(Or, to be more precise... a potential living immortal? A replicon? Whatever.)

A self-replicating physical entity with...

3 Core Components:

  • Quine: Contains the code to produce itself
  • Parser: A parser for that code
    • Code includes instructions for the parser; ideally, compressed instructions
  • Power: Actively running (probably on some substrate)
    • Not actively running is death, although for some the death is temporary.
    • Access to resources may be a subcomponent of this?


... (read more)
deluks917's Shortform

The coronavirus response has been so bad I no longer doubt many Sci-Fi premises. Before I often said to myself "you have tech that can do X and you still have problem Y. Ridiculous!". But apparently, we can make a coronavirus vaccine in two days. But we still had over a year of lockdowns and millions of deaths. One in six hundred people in the USA have died of the virus but we blocked a vaccine because one in a million people who take it MIGHT develop treatable blood clots. 

My standards for 'realistic' dysfunction have gotten a lot lower.

On the flipside: WTF Star Trek? 

3eigen2dThe heads of Government and the FDA don't work like you do. Who knows the incentives that they have? It's entirely possible that for them this is just a political play(moving chess pieces) that make sense for them while the well-being of the people take secondary place to a particular political move. This wouldn't be the first thing that this happens in any kind of government agency, but, at any rate, it's too early to be skeptical. We need to see how this unfolds, may be the pausing don't last as much.
3niplav2dI remember Yudkowsky asking for a realistic explanation [] for why the Empire in Star Wars is stuck in an equilibrium where it builds destroyable gigantic weapons.
Mathisco's Shortform

I once read a comment somewhere that Paul Graham is not a rationalist, though he does share some traits, like writing a lot of self-improvement advice. From what I can tell Paul himself considers himself a builder; a builder of code and companies. But there is some overlap with rationalists, Paul Graham mostly builds information systems. (He is somewhat disdainful of hardware, which I consider the real engineering, but I am a physicist.) Rationalists are focussed on improving their own intelligence and other forms of intelligence. So both spend a great dea... (read more)

Showing 3 of 4 replies (Click to show all)

This is a really good comment. If you care to know more about his thinking, he has a book called, "hackers and painters" which I think sums up very well his views. But yes, it's a redistribution of wealth and power from strong people and bureaucrats to what he calls "nerds" as in people who know technology deeply and actually build things.

The idea of instrumental rationality touches at the edges of builders, and you need to if you ever so desire to act in the world.

1Mathisco11dI'm not sure I follow. Whether it's the evolving configuration of atoms or bits, both can lead to new applications. The main difference to me seems that today it is typically harder to configure atoms than bits, but perhaps that's just by our own design of the atoms underlying the bits? If some desired information system would require a specific atomic configuration, then you'd be hardware constrained again. Let's say that in order to build AGI we find out you actually need super power efficient computronium, and silicon can't do that, you need carbon. Now it's no longer a solved hardware problem, you are going to have to invest massively in carbon based computing. Paul and the rationalists are stuck waiting for the hardware engineers.
1Gerald Monroe11dI am saying below a certain level of abstraction it becomes a solved problem in that you precisely have defined what correctness is and have fully represented your system. And you can trivially check any output and validate it versus a model. The reason software fails constantly is we don't have a good definition that can be checked by computer of what correctness means. Software Unit tests help but are not nearly as reliable as tests for silicon correctness. Moreover software just ends up being absurdly more complex than hardware and ai systems are worse. Part of it is "unique complexity". A big hardware system is millions of copies of the same repeating element. And locality matters - an element cannot affect another one far away unless a wire connects them. A big software system is millions of copies of often duplicated and nested and invisibly coupled code.
Daniel Kokotajlo's Shortform

Years after I first thought of it, I continue to think that this chain reaction is the core of what it means for something to be an agent, AND why agency is such a big deal, the sort of thing we should expect to arise and outcompete non-agents. Here's a diagram:

Roughly, plans are necessary for generalizing to new situations, for being competitive in contests for which there hasn't been time for natural selection to do lots of optimization of policies. But plans are only as good as the knowledge they are based on. And knowledge doesn't come a priori; it nee... (read more)

Showing 3 of 4 replies (Click to show all)
6Yoav Ravid2dSeems similar to the OODA loop []

Yep! I prefer my terminology but it's basically the same concept I think.

4Daniel Kokotajlo2dI disagree; I think we go astray by counting things like thermostats as agents. I'm proposing that this particular feedback loop I diagrammed is really important, a much more interesting phenomenon to study than the more general category of feedback loop that includes thermostats.
lsusr's Shortform

An aircraft carrier costs $13 billion. An anti-ship cruise missile costs $2 million. Few surface warships survived the first day of the Singularity War.

A cruise missile is a complex machine, guided by sensitive electronics. Semiconductor fabricators are even more complex machines. Few semiconductor factories survived the nuclear retaliation.

A B-52 Stratofortress is a simpler machine.

Robert (Bob) Manchester's bomber flew west from Boeing Field. The crew disassembled their landing gear and dropped it in the Pacific Ocean. The staticy voice of Mark Campbell, ... (read more)

MikkW's Shortform

In response to my earlier post about Myers-Briggs (where I suggested a more detailed notation for more nuanced communication about personality types), it was pointed out that there is some correlation between the four traits being measured, and this makes the system communicate less information on average than it otherwise would (The traditional notation would communicate 4 bits, my version would communicate ~9.2 if there was no correlation).

I do object to the characterization that it all measures "the same thing", since none of the traits perfectly predic... (read more)

wunan's Shortform

I have a potential category of questions that could fit on Metaculus and work as an "AGI fire alarm." The questions are of the format "After an AI system achieves task x, how many years will it take for world output to double?"

lsusr's Shortform

I don't mind jumping through a few extra hoops in order to access a website idiosyncratically. But sometimes the process feels overly sectarian.

I was trying out the Tencent cloud without using Tor when I got a CAPTCHA. Sure, whatever. They verified my email. That's normal. Then they wanted to verify my phone number. Okay. (Using phone numbers to verify accounts is standard practice for Chinese companies.) Then they required me to verify my credit card with a nominal $1 charge. I can understand their wanting to take extra care when it comes to processing in... (read more)

I do mind, after having spent several minutes annotating images of self-driving cards

I think it's worst when you have edge cases like the Google Captcha that shows 16 tiles and you have to choose which tiles contain the item they are looking for and some of the tails contain it only a little bit on the edge. 

Charlie Steiner's Shortform

I think you can steelman Ben Goertzel-style worries about near-term amoral applications of AI being bad "formative influences" on AGI, but mostly under a continuous takeoff model of the world. If AGI is a continuous development of earlier systems, then maybe it shares some datasets and learned models with earlier AI projects, and definitely it shares the broader ecosystems of tools, dataset-gathering methodologies, model-evaluating paradigms, and institutional knowledge on the part of the developers. If the ecosystem in which this thing "grows up" is one t... (read more)

MikkW's Shortform

Random thought: if you have a big enough compost pile, would it spontaneously break into flames due to the heat generated by the bioprocesses that occur therein? If so, at what size would it burst into flames? Surely it could happen before it reached the size of the sun, even ignoring gravitational effects.

(Just pondering out loud, not really asking unless someone really wants to answer)

For a value of "break into flames" that matches damp and poorly-oxygenated fuel, yep! This case in Australia is illustrative; you tend to get a lot of nasty smoke rather than a nice campfire vibe.

You'd have to mismanage a household-scale compost pile very badly before it spontaneously combusts, but it's a known and common failure mode for commercial-scale operations above a few tons. Specific details about when depend a great deal on the composition of the pile; with nitrate filmstock it was possible with as little as a few grams.

niplav's Shortform

The child-in-a-pond thought experiment is weird, because people use it in ways it clearly doesn't work for (especially in arguing for effective altruism).

For example, it observes you would be altruistic in a near situation with the drowning child, and then assumes that you ought to care about people far away as much as people near you. People usually don't really argue against this second step, but very much could. But the thought experiment makes no justification for that extension of the circle of moral concern, it just assumes it.

Similarly, it says noth... (read more)

Showing 3 of 4 replies (Click to show all)

Right, my gripe with the argument is that these first two assumptions are almost always unstated, and most of the time when people use the argument, they "trick" people into agreeing with assumption one.

(for the record, I think the first premise is true)

2Dagon7dOn self-reflection, I just plain don't care about people far away as much as those near to me. Parts of me think I should, but other parts aren't swayed. The fact that a lot of the motivating stories for EA don't address this at all is one of the reasons I don't listen very closely to EA advice. I am (somewhat) an altruist. And I strive to be effective at everything I undertake. But I'm not an EA, and I don't really understand those who are.
5habryka7dYep, that's fine. I am not a moral prescriptivist who tells you what you have to care about. I do think that you are probably going to change your mind on this at some point in the next millennium if we ever get to live that long, and I do have a bunch of arguments that feel relevant, but I don't think it's completely implausible you really don't care. I do think that not caring about how people are far away is pretty common, and building EA on that assumption seems fine. Not all clubs and institutions need to be justifiable to everyone.
MikkW's Shortform

Last month, I wrote a post here titled "Even Inflationary Currencies Should Have Fixed Total Supply", which wasn't well-received. One problem was that the point I argued for wasn't exactly the same as what the title stated: I supported both currencies with fixed total supply, and currencies that instead choose to scale supply proportional to the amount of value in the currency's ecosystem, and many people got confused and put off by the disparity between the title and my actual thesis; indeed, one of the most common critiques in the comments was a reiterat... (read more)

Daniel Kokotajlo's Shortform

Productivity app idea:

You set a schedule of times you want to be productive, and a frequency, and then it rings you at random (but with that frequency) to bug you with questions like:

--Are you "in the zone" right now? [Y] [N]

--(if no) What are you doing? [text box] [common answer] [ common answer] [...]

The point is to cheaply collect data about when you are most productive and what your main time-wasters are, while also giving you gentle nudges to stop procrastinating/browsing/daydream/doomscrolling/working-sluggishly, take a deep breath, reconsider your priorities for the day, and start afresh.

Probably wouldn't work for most people but it feels like it might for me.

Showing 3 of 4 replies (Click to show all)
5gjm7d"Are you in the zone right now?" "... Well, I was."

I'm betting that a little buzz on my phone which I can dismiss with a tap won't kill my focus. We'll see.

3NaiveTortoise7dThis is basically a souped up version of TagTime [] (by the Beeminder folks) so you might be able to start with their implementation.
G Gordon Worley III's Shortform

More surprised than perhaps I should be that people take up tags right away after creating them. I created the IFS tag just a few days ago after noticing it didn't exist but wanted to link it and I added the first ~5 posts that came up if I searched for "internal family systems". It now has quite a few more posts tagged with it that I didn't add. Super cool to see the system working in real time!

MikkW's Shortform

I learned to type in Dvorak nearly a decade ago, and any time I have typed on a device that supports it, I have used it since then. I don't know if it actually is any better than QWERTY, but I do notice that I enjoy the way it feels to type in Dvorak; the rhythm and shape of the dance my fingers make is noticeably different from when I type on QWERTY.

Even if Dvorak itself turns out not to be better in some way (fx. speed, avoiding injury, facilitation of mental processes) than QWERTY, it is incredibly unlikely that there does not exist some configuration of keys that is provably superior to QWERTY.

Also, hot take: Colemak is the coward's Dvorak.

Load More