All of skybrian's Comments + Replies

Rising rents and appropriate responses

I don't have citations for you, but it seems relevant that income far in the future gets discounted quite a bit compared to current income, which would imply that short-term incentives are more important than long-term incentives.

(A better argument would need to be made with realistic numbers.)

Rising rents and appropriate responses

Building new hubs doesn't need to be literally building something new.  A lot could be done just by load-balancing with cities that have lower rents and could use the jobs. Suppose that places where growth is a problem cooperated more with places that want more growth?

Place-Based Programming - Part 1 - Places

This method of caching assumes that an expression always evaluates to the same value. This is sometimes true in functional programming, but only if you're careful. For example, suppose the expression is a function call, and you change the function's definition and restart your program. When that happens, you need to delete the out-of-date entries from the cache or your program will read an out-of-date answer.

Also, since you're using the text of an expression for the cache key, you should only use expressions that don't refer to any local variables. For exa... (read more)

Note to readers: skybrian's parent comment was written when this post was titled "Place-Based Programming", before I changed the title to "Place-Based Programming - Part 1". -------------------------------------------------------------------------------- Your are correct that the code here in Part 1 breaks when you use variables with nonlocal scope. I begin to solve this problem in Part 2 [] . Yes. I often think about this project as "writing a compiler". Some of the techniques I use come from Makefiles.
I think this overstates the difficulty, referential transparency is the norm in functional programming, not something unusual. As I understand, this system is mostly useful if you're using it for almost every function. In that case, your inputs are hashes which contain the source code of the function that generated them, and therefore your caches will invalidate if an upstream function's source code changed. Agreed. I agree that it's essentially a framework, and you'd need buy-in from a team in order to consistently use it in a repository. But I've seen teams buy into heavier frameworks pretty regularly; this version seems unusual but not particularly hard to use/understand. It's worth noting that bad caching systems are pretty common in data science, so something like this is potentially a big improvement there.

A model relased on with "GPT" in the name before end of 2022. Could be either GPTX where X is a new name for GPT4, but should be an iteration over GPT-3 and should have at least 10x more parameters.

Politics is way too meta

When you're actually a little curious, you might start by using a search engine to find a decent answer to your question.  At least, if it's the sort of question for which that would work. Maybe even look for a book to read?

But, maybe we should acknowledge that much of the time we aren't actually curious and are just engaging in conversation for enjoyment? In that case, cheering on others who make an effort to research things and linking to their work is probably the best you can do. Even if you're not actually curious, you can notice people who are, ... (read more)

Return to New York City

Museums I'll give you (when they are open again).

For bookstores, in these days of electronic books, I don't think it matters where you live. I remember the last time I went into Powell's. I looked around for a while, dutifully bought one book for old time's sake, and realized later while reading it that I was annoyed that it wasn't electronic. I still go to a local library (when there's not a pandemic) but it's mostly for the walk.

Teachers: that's something I hadn't considered. Since getting out of school, I'm mostly self-taught.

Politics is way too meta

Of course this post is all meta, and my comment will be meta as well. We do it because it's easy.

I think part of the solution is being actually curious about the world.

For me, this perfectly hits the nail on the head. This is a somewhat weird question, but like, how do I do that? I've noticed multiple communities fall into the meta-trap, and even when members notice it can be difficult to escape. While the solution is simply to "stop being meta", that is much harder said than done. When I noticed this happening in a community I am central in organizing I pushed back by bringing my own focus to output instead of process hoping others would follow suit. This has worked somewhat and we're definitely on a better track. I wonder what dynamics lead to this 'death by meta' syndrome, and if there is a cure.
Return to New York City

When enthusiastic New Yorkers say things like "everything at your fingertips" I want to ask what they mean by everything, since it seems subjective, based on what sorts of places one values? In this case: restaurants and parks?

I think the answer is: actually everything, minus a few odds and ends. There are some things that are not available, mainly having to do with physical reality, like: hiking trails, suburban life, buildings as old as in some other places. But if it's something related to human culture, you'll find it. Food is the easiest dimension to talk about because it's everywhere, but if you're looking for art, history, books, NYC has you covered with multiple galleries, a museum it would easily take you a few days to get through, and the 4th largest library in the world. If you are searching for teachers or mentors, you'll find plenty of classes and workshops, including the very best ones. I was surprised that 2 or 3 of the best BJJ gyms in the world are located in Manhattan. So, to answer your question, New York generally has everything you need to satisfy a curiosity.
A whirlwind tour of Ethereum finance

I'm wondering if these loans should really be considered loans, or some other kind of trade? It sounds like you're doing something like trading 100 X for 90 Y and the option to later pay 95 Y for 100 X. Is there any real "defaulting" on the loan? It seems like you just don't exercise the option.

I wonder what “O(n) performance” is supposed to mean, if anything?

We got what's needed for COVID-19 vaccination completely wrong

The question here is whether general arguments that experts make based on inference are reliable, or do you need specific evidence. What is the track record for expert inferences about vaccines?

From a quick search, it seems that the clinical trial success rate for vaccines is about 33%, which is significantly higher than for medical trials in general, but still not all that high? Perhaps there is a better estimate for this.

Estimation of clinical trial success rates and related parameters

Covid 2/4: Safe and Effective Vaccines Aplenty

I found an answer on the PCR question here:

But there is something good to say about their data collection: since the UK study that’s included in these numbers tested its subjects by nasal swab every week, regardless of any symptoms, we can actually get a read on something that everyone’s been wondering about: transmission.

Covid 2/4: Safe and Effective Vaccines Aplenty

AstraZeneca has not applied for emergency use authorization, because it has been told not to do so.


That resolves a mystery for me if true. How do you know this?

(I was wondering if maybe they are selling all they can make in other countries.)

After reading the article i'm wondering why they aren't selling all they can make in other countries.
This is how the process works, and also it leaks. AZ has been told that FDA will not approve without the USA data. I don't have links handy but it's very clear.
Covid 2/4: Safe and Effective Vaccines Aplenty

I'm not sure about this statement in the blog post:

In the meantime, the single dose alone is 76% effective, presumably against symptomatic infection (WaPo) and was found to be 67% effective against further transmission.

I read another article saying that this is disputed by some experts:

With a seductive number, AstraZeneca study fueled hopes that eclipsed its data

Media reports seized on a reference in the paper from Oxford researchers that a single dose of the vaccine cut positive test results by 67%, pointing to it as the first evidence that a vaccine coul

... (read more)
I found an answer on the PCR question here [] :
That piece seems like pure FUD and "no evidence" to me. I don't think this is "experts disagree" in a meaningful way.
No Causation without Reification

What’s an example of a misconception someone might have due to having a mistaken understanding of causality, as you describe here?

2G Gordon Worley III2y
Generally, supposing the existence of particular things prior to the experience of them. The key insight is to see that the existence of "things" is not identical to the existence of reality out of which things are carved. Take literally anything and it's your example: a cup, an atom, experience, causation, dancing, etc. You can find none of these things in the territory itself, only in your understanding of it (and yet something is there in the territory for you to create a useful understanding of it, but it only becomes a thing by virtue of some perception of it).
The bads of ads

This is a bizarre example, sort of like using Bill Gates to show why nobody needs to work for a living. It ignores the extreme inequality of fame.

Tesla doesn’t need advertising because they get huge amounts of free publicity already, partly due to having interesting, newsworthy products, partly due to having a compelling story, and partly due to publicity stunts.

However, this free publicity is mostly unavailable for products that are merely useful without being newsworthy. There are millions of products like this. An exciting product might not need adverti... (read more)

There are many reasons why a customer might buy a product. I might buy one kind of product because it's cheaper then the next. I might buy one kind of product because of publicity stunts. I might buy a product because a friend recommended it to me because they had great experiences with it. I might by a product because it has good reviews. I might buy a product because it has good advertising. On the other side an executive is thinking "What's my core strategy for aquiring customers?" If the core strategy is advertising and not producing products with good value propositions, that's to me a bad signal. 80/20 thinking does mean that many times there's a core strategy on which a company focuses.
Babble & Prune Thoughts

It seems like some writers have habits to combat this, like writing every day or writing so many words a day. As long as you meet your quota, it’s okay to try harder.

Some do this in public, by publishing on a regular schedule.

If you write more than you need, you can prune more to get better quality.

I enjoyed the book write better, faster, in which an author set out on a series of self-experimentations to write faster. First she tried measuring words per hour. She was quite successful at getting this to be much higher, but it turned out that this resulted in writing for less time each day (so average wordcount per day was about the same). She then tried to maximize words per day, which was again successful, but this similarly resulted in writing less on subsequent days. (She might have then had the same experience on the week level, I don't remember.)... (read more)

Exposure or Contacts?

One aspect that might be worth thinking about is the speed of spread. Seeing someone once a week means that it slows down the spread by 3 1/2 days on average, while seeing them once a month slows things down by 15 days on average. It also seems like they are more likely to find out they have it before they spread it to you?

GPT-3, belief, and consistency

Yes, sometimes we don't notice. We miss a lot. But there are also ordinary clarifications like "did I hear you correctly" and "what did you mean by that?" Noticing that you didn't understand something isn't rare. If we didn't notice when something seems absurd, jokes wouldn't work.

GPT-3, belief, and consistency

It's not quite the same, because if you're confused and you notice you're confused, you can ask. "Is this in American or European date format?" For GPT-3 to do the same, you might need to give it some specific examples of resolving ambiguity this way, and it might only do so when imitating certain styles.

It doesn't seem as good as a more built-in preference for noticing and wanting to resolve inconsistency? Choosing based on context is built in using attention, and choosing randomly is built in as part of the text generator.

It's also worth noticing that the GPT-3 world is the corpus, and a web corpus is a inconsistent place.

You can if you do, but most people never notice and those who notice some confusion are still blissfully ignorant of the rest of their self-contradicting beliefs. And by most people I mean you, me and everyone else. In fact, if someone pointed out a contradictory belief in something we hold dear, we would vehemently deny the contradiction and rationalize it to no end. And yet we consider ourselves believing something. If anything, GPT-3's beliefs are more belief-like than those of humans.
10/50/90% chance of GPT-N Transformative AI?

Having demoable technology is much different than having reliable technology. Take the history of driverless cars. Five teams completed the second DARPA grand challenge in 2005. Google started development secretly in 2009 and announced the project in October 2010. Waymo started testing without a safety driver on public roads in 2017. So we've had driverless cars for a decade, sort of, but we are much more cautious about allowing them on public roads.

Unreliable technologies can be widely used. GPT-3 is a successor to autocomplete, which everyone alrea... (read more)

Where do people discuss doing things with GPT-3?

In that case, I'm looking for people sharing interesting prompts to use on AI Dungeon.

1Michael Norton2y
If that's what you're really interested in, then join the AI Dungeon Discord server: []
Where do people discuss doing things with GPT-3?

Where is this? Is it open to people who don't have access to the API?

2Michael Norton2y
Yes it is; I don't remember where I found this link but it was definitely somewhere public. []
No, and that's why it's good, because it's the actual API users, and not people punching in things to AI Dungeon and wondering why they don't get the same results...
GPT-3 Gems

I'm suggesting something a little more complex than copying. GPT-3 can give you a random remix of several different clichés found on the Internet, and the patchwork isn't necessarily at the surface level where it would come up in a search. Readers can be inspired by evocative nonsense. A new form of randomness can be part of a creative process. It's a generate-and-test algorithm where the user does some of the testing. Or, alternately, an exploration of Internet-adjacent story-space.

It's an unreliable narrator and I suspect it will be an unreliable search engine, but yeah, that too.

Replicating the replication crisis with GPT-3?

I was making a different point, which is that if you use "best of" ranking then you are testing a different algorithm than if you're not using "best of" ranking. Similarly for other settings. It shouldn't be surprising that we see different results if we're doing different things.

It seems like a better UI would help us casual explorers share results in a way that makes trying the same settings again easier; one could hit a "share" button to create a linkable output page with all relevant settings.

It could also save the alternate responses that either the u... (read more)

Replicating the replication crisis with GPT-3?

I don't see documentation for the GPT-3 API on OpenAI's website. Is it available to the public? Are they doing their own ranking or are you doing it yourself? What do you know about the ranking algorithm?

It seems like another source of confusion might be people investigating the performance of different algorithms and calling them all GPT-3?

The current docs do seem to be behind the login wall. (They're integrated with your API token to make copy-paste easier, so that's not too surprising.) It's also true that people have been using different algorithms, but regular API users are typically clear if they're not using davinci and confusion is mostly the fault of AI Dungeon users: we don't know what AID does, and AID users sometimes don't even pick the right model option and still say they are using "GPT-3".
Replicating the replication crisis with GPT-3?

How do you do ranking? I'm guessing this is because you have access to the actual API, while most of us don't?

On the bright side, this could be a fun project where many of us amateurs learn how to do science better, but the knowledge of how to do that isn't well distributed yet.

Yes. I don't think AID exposes ranking. (If they pay per API call, doing best-of=n would be n times more expensive, and for creative uses like AID, ranking/best-of is not that useful and is certainly not n times better. Very diminishing returns there - unless you're asking tricky or difficult questions, where ranking often seems to hit on the right answer where regular GPT-3 fails. See also the Meena paper on how much ranking improved over baseline Meena.)
GPT-3 Gems

We take the web for granted, but maybe we shouldn't. It's very large and nobody can read it all. There are many places we haven't been that probably have some pretty good writing. I wonder about the extent to which GPT-3 can be considered a remix of the web that makes it seem magical again, revealing aspects of it that we don't normally see? When I see writing like this, I wonder what GPT-3 saw in the web corpus. Is there an archive of Tolkien fanfic that was included in the corpus? An undergrad physics forum? Conversations about math and computer science?

Such as a darknet marketplace where animals can trade valuable resources for electricity? :D But yeah, I agree, if there are places debating a topic that resembles the prompt, GPT-3 could be good at including them in the debate. So maybe if the result is too good, it makes sense to check parts of it by a search engine. Maybe it would even make sense to use GPT-3 purposefully to search for something on the internet. Like, if you have a vague suspicion that something could exist, but you don't know the right keywords to type into the search engine, maybe you could just describe the thing, and hope that GPT-3 finds the right words and tells you something that you can search later. Not sure if this actually would work.
To what extent is GPT-3 capable of reasoning?

Rather than putting this in binary terms (capable of reason or not), maybe we should think about what kinds of computation could result in a response like this?

Some kinds of reasoning would let you generate plausible answers based on similar questions you've already seen. People who are good at taking tests can get reasonably high scores on subjects they don't fully comprehend, basically by bluffing well and a bit of luck. Perhaps something like that is going on here?

In the language of "Thinking, Fast and Slow", this might be "Syst... (read more)

To what extent is GPT-3 capable of reasoning?
Answer by skybrianJul 21, 2020Ω14

GPT-3 has partially memorized a web corpus that probably includes a lot of basic physics questions and answers. Some of the physics answers in your interview might be the result of web search, pattern match, and context-sensitive paraphrasing. This is still an impressive task but is perhaps not the kind of reasoning you are hoping for?

From basic Q&A it's pretty easy to see that GPT-3 sometimes memorizes not only words but short phrases like proper names, song titles, and popular movie quotes, and probably longer phrases if they are common enough.

Google's Q&A might seem more magical too if they didn't link to the source, which gives away the trick.

How should I modify the problems I gave it? What would be the least impressive test which would convince you it is reasoning, and not memorizing? (Preferably something that doesn't rely on eg rhyming, since GPT-3 uses an obfuscating input encoding)
5Daniel Kokotajlo2y
GPT-3 is still capable of reasoning if some of the answers were copied from the web. What you need for it to not be capable of reasoning is for all of the answers to have been copied from the web. Given its ability to handle random weird hypotheticals we just thought up, I'm pretty convinced at this point that it isn't just pulling stuff from the web, at least not all the time.
What will the economic effects of COVID-19 be?

This is more about expanding the question with slightly more specific questions:

Currently it seems like there are many people who are not scared enough, but I wonder if sentiment could quickly go the other way?

A worst-case scenario for societal collapse is that some "essential" workers are infected and others decide that it is too risky to keep working, and there are not enough people to replace them. Figuring out which sectors might be most likely to have critical labor shortages seems important.

An example of a "labor" shortage might b... (read more)

Frivolous speculation about the long-term effects of coronavirus

Yeah, I don't see it changing that drastically; more likely it will be a lot of smaller and yet significant changes that make old movies look dated. Something like how the airports changed after 9/11, or more trivially, that time when all the men in America stopped wearing hats.

Crisis and opportunity during coronavirus

I'm wondering what's a way to keep better tabs on what people are talking in the rationalist community without reading everything? There is a lot of speculation, but sometimes very useful signal.

I feel like I'm reasonably in touch from reading Slate Star Codex and occasionally checking in here, and yet the first thing I saw that really got my attention was "Seeing the Smoke" getting posted on Hacker News. I guess I'm not following the right people yet?

Assembling Sets for Contra

I'm wondering if anyone can recommend some recordings that they like on YouTube or Spotify of this sort of music? I don't know if I've heard it before.

YouTube: Great Bear Trio: Benton's Hallucinations [] Spotify: Buddy System [] , Great Bear [] , Mean Lids [] , Elixir [] and my band, Free Raisins []
[Meta] New moderation tools and moderation guidelines

I'm just a lurker, but as an FYI, on The Well, hidden comments were marked <hidden> (and clickable) and deleted comments were marked <scribbled> and it seemed to work out fine. I suppose with more noise, this could be collapsed to one line: <5 scribbled>.

Security Mindset and the Logistic Success Curve

I mean things like using mathematical proofs to ensure that Internet-exposed services have no bugs that a hostile agent might exploit. We don't need to be able to build an AI to improve defences.

Security Mindset and the Logistic Success Curve

I think odds are good that, assuming general AI happens at all, someone will build a hostile AI and connect it to the Internet. I think a proper understanding the security mindset is that the assumption "nobody will connect a hostile AI to the Internet" is something we should stop relying on. (In particular, maintaining secrecy and internatonal cooperation seems unlikely. We shouldn't assume they will work.)

We should be looking for defenses that aren't dependent of the IQ level of the attacker, similar to how mathematical proofs are ind... (read more)

3Rob Bensinger5y
Yup, all of this seems like the standard MIRI/Eliezer view. I don't know what the relevance of "mathematical proofs" is. Are you talking about applying formal methods of some kind to the problem of ensuring that AGI technology doesn't leak, and saying that AGI is unnecessary for this task? I'm guessing that part of the story you're missing is that proliferation of AGI technology is at least as much about independent discovery as it is about leaks, splintering, or espionage. You have to address those issues, but the overall task of achieving nonproliferation is much larger than that, and it doesn't do a lot of good to solve part of the problem without solving the whole problem. AGI is potentially a route to solving the whole problem, not to solving the (relatively easy, though still very important) leaks/espionage problem.
Security Mindset and the Logistic Success Curve

Even if there's no "friendly part," it seems unlikely that someone who learns the basic principles behind building a friendly AI will be unable to build an unfriendly AI by accident. I'm happy that we're making progress with safe languages, but there is no practical programming language in which it's the least bit difficult to write a bad program.

It would make more sense to assume that at some point, a hostile AI will get an Internet connection, and figure out what needs to be done about that.

Security Mindset and the Logistic Success Curve

I'm happy to see a demonstration that Eliezer has a good understanding of the top-level issues involving computer security.

One thing I wonder though, is why making Internet security better across the board isn't a more important goal in the rationality community? Although very difficult (for reasons illustrated here), it seems immediately useful and also a good prerequisite for any sort of AI security. If we can't secure the Internet against nation-state level attacks, what hope is there against an AI that falls into the wrong hands?

In parti... (read more)

There's no "friendly part" in an AGI in the same way there's no "secure part" in an OS. The kind of friendliness and security we want is deep in the architecture. Most hostile actors also don't want an AGI that kills them. They might still do nasty things with the AGI by giving it bad goals but that's not the core what the AGI argument is about. As far as removing misbehavior modes, that's done in security circles. Critical computers get airgapped to prevent them from getting hacked. In the quest of getting more secure Mozilla build a new programming language that prevents a class of errors that C had.
7Rob Bensinger5y
This is correct. Any reasonable AGI development strategy must have strong closure and security measures in place to minimize the risk of leaks, and deployment has to meet the conditions in Dewey (2014) []. This is also correct, if the idea is to ensure that developers understand their system (and safety-critical subsystems in particular) well enough for the end product to be "friendly"/"aligned." If you're instead saying that alignment isn't a good target to shoot for, then I'm not sure I understand what you're saying. Why not? How do we achieve good long-term outcomes without routing through alignment?
Open thread, November 13 - November 20, 2017

Thanks! Bug filed. Regarding the Intercom chat bubble, I did post one comment a while back (accidentally in the wrong chat room for Lesswrong), but got no response, and I don't see any other responses in either chat room. Also, the indicator always says "away". To the naive user it looks abandoned. Are you sure it's working? Maybe the old chat room should be deleted?

Open thread, November 13 - November 20, 2017

Where do we report bugs? For example, I was unable to leave a comment here using Chrome on an Android tablet. (Desktop is okay.)

Also, is source available? I might be able to make suggestions.

I've been having this issue too! Please report it!
Both on Github: [] There should also be a small chat bubble in the bottom right corner with a welcome message, instructions on how to report bugs and the ability to ping me and Ben directly. Which is often a good choice for situations where you are unsure how you would phrase or construct a Github issue.
Living in an Inadequate World

I'd like to see citations for the claims about maganese and selenium.

A bit of poking around later ... (because I had been taking some Manganese to help with some bone problems. The recomended intake is maybe 1.5mg/day (adult male, more for pregnant women)). The toxic level is around 11mg/day. Among other things it can be neurotoxic. So it has pretty tight margins, similar to, say Fluorine. See links from the wikipedia article. >(EY) You get all the manganese you need from ordinary drinking water, if it hasn’t been distilled or bottled. This really depends. Amounts in drinking water vary wildly from hardly any to far too much. Brown staining on porcelain is one sign that your water is over 0.5mg/Liter, and the taste becomes 'undesirable' at that point too. See and "Intellectual Impairment in School-Age Children Exposed to Manganese from Drinking Water" on (LW mangles the URL sorry) Groundwater or runoff water affected by pollution seems to make it much more likely Mn is too high.
The Journal of High Standards

Prize money helps, but you'd also need to find relevant experts who know enough about each sub-field to tell whether the standards are indeed high. (Usually they are called "judges," but perhaps we could call them "peers?")

It might help to narrow the question: instead of looking for "high standards" (which is vague), the prizes could be awarded based on whether papers already published elsewhere appear to use good statistics. Then you'd only need reviewers who are experts in statistics.

You don't need to look at any paper to decide whether or not the CONSORT standards are high standards. The idea of a standard is that the standard isn't different in every particular case. In medical sciences peer reviewers don't seem to be interested in doing something simple as checking whether the results that a paper reports matches the one's that were preregistered. They do suggest various changes to papers but there's no standard involved in those suggestions but individual opinion of what style individual reviewers like. Letting people in the same subfield review leads to an academic climate where papers are only written to be read by other people of the same subfield. The incentives that reviewers in the present system have in the present system are bad. If you ask the average scientists about the quality of the peer review his papers get, they don't think very highly of it.
The Craft & The Community - A Post-Mortem & Resurrection

From an outside (but sympathetic) perspective, seems like this post would have been better if you started out with "Why we're starting a new rationalist community in Manchester" and took it from there? As it is, I wonder how many people made it to the end.

Against naming things, and so on

I'm reminded of the Oblique Strategies playing cards. Obviously the cards don't provide any sort of rigor. But having them around might be useful for thinking creatively. Might the same apply for Less Wrong jargon?

Moloch's Toolbox (1/2)

Looks like there is a detailed Wiki page about this.

Moloch's Toolbox (1/2)

Yes, everything is terrible. But it seems like, if you're writing a book and discover something like the Omegaven story, it might be worth writing a blog post just about that and seeing if it can get some publicity via social media? (I settled for resharing the 2013 NBC article.)

Inadequacy and Modesty

Maybe compare with epistemic learned helplessness?