All of FactorialCode's Comments + Replies

Another alternative is to use a 440nm light source and a frequency doubling crystal although the efficiency is questionable, there are also other variations based on frequency quadrupling

That said, you can hide it in your user-settings.

This solves my problem, thank you. Also it does look just like the screenshot, no problems other than what I brought up when you click on it.

This might just be me, but I really hate the floating action button on LW. It's an eyesore on what is otherwise a very clean website. The floating action button was designed to "Represent the primary action on a screen" and draw the user's attention to itself. It does a great job at it, but since "ask us anything, or share your feedback" is not the primary thing you'd want to do, it's distracting.

Not only does it do that, but it also give the impression that this is another cobbled together Meteor app and therefore my brain instantly makes me associate it ... (read more)

People in fact use it quite a bit. That said, you can hide it in your user-settings. Also, doublechecking if by default it appears as anything than a subdued circle, like this? (Occasionally we get reports of it being much-more-obtrusive, by showing more messages or something immediately. If it's doing that for you, apologies. That is a bug)
I turned it off long ago, and had forgotten it exists.

I bet this is a side effect of having a large pool of bounded rational agents that all need to communicate with each other, but not necessarily frequently. When two agents only interact briefly, neither agent has enough data to work out what the "meaning" of the other's words. Each word could mean too many different things. So you can probably show that under the right circumstances, it's beneficial for agents in a pool to have a protocol that maps speech-acts to inferences the other party should make about reality (amongst other things, such as other acti... (read more)

I personally think that something more akin to minimum utilitarianism is more inline with my intuitions. That is, to a first order approximation, define utility as (soft)min(U(a),U(b),U(c),U(d)...) where a,b,c,d... are the sentients in the universe. This utility function mostly captures my intuitions as long as we have reasonable control over everyone's outcomes, utilities are comparable, and the number of people involved isn't too crazy.

Money makes the world turn and it enables research, be it academic or independent. I would just focus on getting a bunch of that. Send out 10x to 20x more resumes than you already have, expand your horizons to the entire planet, and put serious effort into prepping for interviews.

You could also try getting a position at CHAI or some other org that supports AI alignment PhDs, but it's my impression that those centres are currently funding constrained and already have a big list of very high quality applicants, so your presence or absence might not make that... (read more)

I'd put my money on lowered barriers to entry on the internet and eternal September effects as the primary driver of this. In my experience the people I interact with IRL haven't really gotten any stupider. People can still code or solve business problems just as well as they used to. The massive spike in stupidity seems to have occurred mostly on the internet.

I think this is because of 2 effects that reinforce each other in a vicious cycle.

  1. Barriers to entry on the internet have been reduced. A long time ago you needed technical know how to even operate

... (read more)

At its core, this is the main argument why the Solomonoff prior is malign: a lot of the programs will contain agents with preferences, these agents will seek to influence the Solomonoff prior, and they will be able to do so effectively.

Am I the only one who sees this much less as a statement that the Solomonoff prior is malign, and much more a statement that reality itself is malign? I think that the proper reaction is not to use a different prior, but to build agents that are robust to the possibility that we live in a simulation run by influence seeking malign agents so that they don't end up like this.

Hmm, at this point it might be just a difference of personalities, but to me what you're saying sounds like "if you don't eat, you can't get good poisoning". "Dual identity" doesn't work for me, I feel that social connections are meaningless if I can't be upfront about myself.

That's probably a good part of it. I have no problem hiding a good chunk of my thoughts and views from people I don't completely trust, and for most practical intents and purposes I'm quite a bit more "myself" online than IRL.

But in any case there will many subnetworks in the net

... (read more)

So first of all, I think the dynamics of surrounding offense are tripartite. You have the the party who said something offensive, the party who gets offended, and the party who judges the others involved based on the remark. Furthermore, the reason why simulacra=bad in general is because the underlying truth is irrelevant. Without extra social machinery, there's no way to distinguish between valid criticism and slander. Offense and slander are both symmetric weapons.

This might be another difference of can try to come up with a differe

... (read more)

I'm breaking this into a separate thread since I think it's a separate topic.

Second, specifically regarding Crocker's rules, I'm not their fan at all. I think that you can be honest and tactful at the same time, and it's reasonable to expect the same from other people.

So I disagree. Obviously you can't impose Croker's rules on others, but I find it much easier and far less mentally taxing to communicate with people I don't expect to get offended. Likewise, I've gained a great deal of benefit from people very straightforwardly and bluntly calling me out... (read more)

5Vanessa Kosoy2y
This might be another difference of personalities, maybe Crocker's rules make sense for some people. The problem is, different people have conflicting interests. If we all had the same utility function then, sure, communication would be only about conveying factual information. But we don't. In order to cooperate, we need not only to share information, but also reassure each other we are trustworthy and not planning to defect. If someone criticizes me in a way that disregards tact, it leads me to suspect that eir agenda is not helping me but undermining my status in the group. You can say, we shouldn't do that, that's "simulacra" and simulacra=bad. But the game theory is real, and you can't just magic it away by wishing it would be different. You can try just taking on faith that everyone are your allies, but then you'll get exploited by defectors. Or you can try to come up with a different set of norms that solves the problem. But that can't be Crocker's rules, at least it can't be only Crocker's rules. Now, obviously you can go too far in the other direction and stop conveying meaningful criticism, or start dancing around facts that need to be faced. That's also bad. But the optimum is in the middle, at least for most people.

First, when Jacob wrote "join the tribe", I don't think ey had anything as specific as a rationalist village in mind? Your model fits the bill as well, IMO. So what you're saying here doesn't seem like an argument against my objection to Zack's objection to Jacob.

So my objection definitely applies much more to a village than less tightly bound communities, and Jacob could have been referring to anything along that spectrum. But I brought it up because you said:

Moreover, the relationships between them shouldn't be purely impersonal and intellectual. A

... (read more)
2Vanessa Kosoy2y
Hmm, at this point it might be just a difference of personalities, but to me what you're saying sounds like "if you don't eat, you can't get good poisoning". "Dual identity" doesn't work for me, I feel that social connections are meaningless if I can't be upfront about myself. I guess? But in any case there will many subnetworks in the network. Even if everyone adopt the "village" model, there will be many such villages.

IMO, F*** or F!#@, I feel like it has more impact that way. Since it means you went out of your way to censor yourself, and it's not just a verbal habit, as would be the case with either fuck or a euphemism.

Is "some of us" more than Wei Dai? Because it seems to me that only Wei Dai is mentioned as an example but it is implied that more people profited - not only by you, but in general when I see that claim.

there are two sides to an options contract, when to buy and when to sell. Wei Dai did well on the first half but updated in the comments on losing most of the gains on the second half. This isn't a criticism, it's hard.

So full disclosure, I'm on the outskirts of the rationality community looking inwards. My view of the situation is mostly filtered through what I've picked up online rather than in person.

With that said, in my mind the alternative is to keep the community more digital, or something that you go to meetups for, and to take advantage of societies' existing infrastructure for social support and other things. This is not to say we shouldn't have strong norms, the comment box I'm typing this in is reminding me of many of those norms right now. But the overall ... (read more)

I think you are both right about important things, and the problem is whether we can design a community that can draw benefits of mutual support in real life, while minimising the risks. Keeping each other at internet distance is a solution, but I strongly believe it is far from the best we can do.

We probably need to accept that different people will have different preferences about how strongly involved they want to become in real life. For some people, internet debate may be the optimal level of involvement. For other people, it would be something more l... (read more)

3Vanessa Kosoy2y
First, when Jacob wrote "join the tribe", I don't think ey had anything as specific as a rationalist village in mind? Your model fits the bill as well, IMO. So what you're saying here doesn't seem like an argument against my objection to Zack's objection to Jacob. Second, specifically regarding Crocker's rules, I'm not their fan at all. I think that you can be honest and tactful at the same time, and it's reasonable to expect the same from other people. Third, sure, social and economic dependencies can create problems, but what about your social and economic dependencies on non-rationalists? I do agree that dilution [] is a real danger (if not necessarily an insurmountable one). I will probably never have the chance to live in a rationalist village, so for me the question is mostly academic. To me, a rationalist village sounds like a good idea in expectation (for some possible executions), but the uncertainty is great. However, why not experiment? Some rationalists can try having their own village. Many others wouldn't join them anyway. We would see what comes out of it, and learn.
This feels like an incredibly important point, the pressures when "the rationalists" are friends your debate with online vs when they are close community you are dependant on.

Sure, tribes also carry dangers such as death spirals and other toxic dynamics. But the solution isn't disbanding the tribe, that's throwing away the baby with the bathwater.

I think we need to be really careful with this and the dangers of becoming a "tribe" shouldn't be understated w.r.t our goals. In a community focused on promoting explicit reason, it becomes far more difficult to tell apart those who are carrying out social cognition from those who are actually carrying out the explicit reason, since the object level beliefs and their justifications... (read more)

7Ben Pace2y
It is definitely the case, especially in the EA community, that I'm surrounded by a lot more people who express alliance via signaling and are making nontrivial commitments, for whom I've not seen real evidence that they understand how to think for themselves or take right action without a high status person telling them to do it. That said I don't find it too hard myself to distinguish between such people, and people where I can say "Yeah, I've seen them do real things".

The problems you discuss are real, but I don't understand what alternative you're defending. The choice is not having society or not having society. You are going to be part of some society anyway. So, isn't it better if it's a society of rationalists? Or do you advocate isolating yourself from everyone as much as possible? I really doubt that is a good strategy.

In practice, I think LessWrong has been pretty good at establishing norms that promote reason, and building some kind of community around them. It's far from perfect, but it's quite good compared t... (read more)

Another option not discussed is to control who your message reaches in the first place, and in what medium. I'll claim, without proof or citation, that social media sites like twitter are cesspits that are effectively engineered to prevent constructive conversation and to exploit emotions to keep people on the website. Given that, a choice that can mitigate these kind of situations is to not engage with these social media platforms in the first place. Post your messages on a blog under your own control or a social media platform that isn't designed to hijack your reward circuitry.

I think you're missing an option, though. You can specifically disavow and oppose the malicious actions/actors, and point out that they are not part of your cause, and are actively hurting it. No censorship, just clarity that this hurts you and the cause. Depending on your knowledge of the perpetrators and the crimes, backing this up by turning them or actively thwarting them may be in scope as well.

There is a practical issue with this solution in the era of modern social media. Suppose you have malicious actors who go on to act in your name, but you... (read more)

To be clear, "stand down" is not condemning. "F them and their destructive actions" is condemning. In more formal settings, "I do not support X, and I do not want anything to do with people doing X". A few examples of clear condemnation being used against someone, where that retaliation is worse than the implied association of doing nothing, would help me understand your comment. Note that If they're not ALREADY associated with you in some way (through their actions and publicity, referencing your reputation without your consent), you don't need to respond in any way. That's a pretty easy option 4, I think.

Assuming we can solve the relevant ethical dilemmas. There is exactly one thing I want:

Waifus. Sentient optimized and personalized romantic partners.

I haven't actually figured that out yet, but several people in this thread have proposed takeaways. I'm leaning towards "social engineering is unreasonably effective". That or something related to keeping a security mindset.

I personally feel that the fact that it was such an effortless attempt makes it more impressive, and really hammers home the lesson we need to take away from this. It's one thing to put in a great deal of effort to defeat some defences. It's another to completely smash through them with the flick of a wrist.

What exactly do you think "the lesson we need to take away from this" is? (Feel free to just link if you wrote that elsewhere in this comment section)

Props to whoever petrov_day_admin_account was for successfully red-teaming lesswrong.

Well, they did succeed, so for that they get points, but I think it was more due to a very weak defense on behalf of the victim rather than a very strong effort by petrov_day_admin_account. 

Like, the victim could have noticed things like: 
* The original instructions were sent over email + LessWrong message, but the phishing attempt was just LessWrong
* The original message was sent by Ben Pace, the latter by petrov_day_admin_account
* They were sent at different points in time, the latter of which was more correlated by the FB post that caused the ... (read more)

Agreed, this is probably the best lesson of all.  If the buttons exist, they can be hacked or the decision makers can be socially engineered.

270 people might have direct access, but the entire world has indirect access.

As much as I hate to say it, I don't think that it makes much sense for the main hub of the rationalist movement to move away from Berkeley and the Bay Area. There are several rationalist adjacent organizations that are firmly planted in Berkely. The ones that are most salient to me are the AI and AI safety orgs. You have OpenAI, MIRI, CHAI, BAIR, etc. Some of these could participate in a coordinated move, but others are effectively locked in place due to their tight connections with larger institutions.

I think that more creative options need to be brainstormed and explored to deal with the situation in Berkley.

Ehh, Singapore is a good place to do business and live temporarily. But mandatory military service for all male citizens and second gen permanent residents, along with the work culture make it unsuitable as a permanent location to live. Not to mention that there's a massive culture gap between the rats and the Singaporeans.

I think the cooperative advantages mentioned here have really been overlooked when it comes to forecasting AI impacts, especially in slow takeoff scenarios. A lot of forecasts, like what WFLL, mainly posit AI's competing with each other. Consequently Molochian dynamics come into play and humans easily lose control of the future. But with these sorts of cooperative advantages, AIs are in an excellent position to not be subject to those forces and all the strategic disadvantages they bring with them. This applies even if an AI is "merely" at the human level.... (read more)

2Rohin Shah2y
For the most part, to the extent that we will have these advantages, it still doesn't suggest a discontinuity; it suggests that we will be able to automate tasks with weaker / less intelligent AI systems than you might otherwise have thought. I usually think of Part 1 of WFLL happening prior to reaching what I would call human level AI, because of these AI advantages. Though the biggest AI advantage feeding into this is simply that AI systems can be specialized to particular tasks, whereas humans become general reasoners and then apply their general reasoning to particular tasks.

Just use bleeding edge tech to analyze ancient knowledge from the god of information theory himself.

This paper seems to be a good summary and puts a lower bound on entropy of human models of english somewhere between 0.65 and 1.10 BPC. If I had to guess, the real number is probably closer 0.8-1.0 BPC as the mentioned paper was able to pull up the lower bound for hebrew by about 0.2 BPC. Assuming that regular english compresses to an average of 4* tokens per character, GPT-3 clocks in at 1.73/ln(2)/4 = 0.62 BPC. This is lower than the lower bound mentioned... (read more)

From that paper: > A new improved method for evaluation of both lower and upper bounds of the entropy of printed texts is developed. "Printed texts" probably falls a standard deviation or three above the median human's performance. It's subject to some fairly severe sampling bias.
3Daniel Kokotajlo2y
Hmmm, your answer contradicts Gwern's answer. I had no idea my question would be so controversial! I'm glad I asked, and I hope the controversy resolves itself eventually...

I'm OOTL, can someone send me a couple links that explain the game theory that's being referenced when talking about a "battle of the sexes"? I have a vague intuition from the name alone, but I feel this is referencing a post I haven't read.


I'm gonna go with barely, if at all. When you wear a surgical mask and you breath in, a lot of air flows in from the edges, without actually passing through the mask, so the mask doesn't have very good opportunity to filter the air. At least with N95 and N99 mask, you have a seal around your face, and this forces the air through the filter. Your probably better off wearing a wet bandana or towel that's been tied in such a way as to seal around your face, but that might make it hard to breath.

I found this, which suggests that they're generally ineffective. ... (read more)

Yeah, I'll second the caution to draw any conclusions from this. Especially because this is macroeconomics.

It is my understanding that this is broadly correct. It is also my understanding that this is not common knowledge.

Careful with that. It's an accounting identity, and those have a reputation in economics for being easily misleading. Terms in accounting identities are usually aggregates of other terms with lots of interactions which are nonobvious from the identity.
Thanks, that is exactly the article I needed! I knew my question was at least MMT-adjacent, I never made the accounting identity connection though.

One hypothesis I have is that even in the situation where there is no goal distribution and the agent has a single goal, subjective uncertainty makes powerful states instrumentally convergent. The motivating real world analogy being that you are better able to deal with unforeseen circumstances when you have more money.

I've gone through a similar phase. In my experience you eventually come to terms with those risks and they stop bothering you. That being said, mitigating x and s-risks has become one of my top priorities. I now spend a great deal of my own time and resources on the task.

I also found learning to meditate helps with general anxiety and accelerates the process of coming to terms with the possibility of terrible outcomes.

The way I was envisioning it is that if you had some easily identifiable concept in one model, e.g. a latent dimension/feature that corresponds to the log odd of something being in a picture, you would train the model to match the behaviour of that feature when given data from the original generative model. Theoretically any loss function will do as long as the optimum corresponds to the situation where your "classifier" behaves exactly like the original feature in the old model when both of them are looking at the same data.

In practice though, we're compu... (read more)

I think you can loosen (b) quite a bit if you task a separate model with "delineating" the concept in the new network. The procedure does effectively give you access to infinite data, so the boundary for the old concept in the new model can be as complicated as your compute budget allows. Up to and including identifying high level concepts in low level physics simulations.

We currently have no criteria by which to judge the performance of such a separate model. What do we train it to do, exactly? We could make up some ad-hoc criterion, but that suffers from the usual problem of ad-hoc criteria: we won't have a reliable way to know in advance whether it will or will not work on any particular problem or in any particular case.

I think the eventual solution here (and a major technical problem of alignment) is to take an internal notion learned by one model (i.e. found via introspection tools), back out a universal representation of the real-world pattern it represents, then match that real-world pattern against the internals of a different model in order to find the "corresponding" internal notion.

Can't you just run the model in a generative mode associated with that internal notion, then feed that output as a set of observations into your new model and see what lights up in i... (read more)

This works if both (a) both models are neural nets, and (b) the "concept" cleanly corresponds to one particular neuron. You could maybe loosen (b) a bit, but the bottom line is that the nets have to represent the concept in a particular way - they can't just e.g. run low-level physics simulations in order to make predictions. It would probably allow for some cool applications, but it wouldn't be a viable long-term path for alignment with human values.

I think this is pretty straight forward to test. GPT-3 gives joint probabilities of string continuations given context strings.

Step 1: Give it 2 promps, one suggesting that it is playing the role of a smart person, and one where it is playing the roll of a dumb person.

Step 2: Ask the "person" a question that demonstrates that persons intelligence. (something like a math problem or otherwise)

Step 2: Write continuations where the person answers correctly and incorrectly

Step 3: Compare the relative probabilities GPT-3 assigns to each continuation given the p... (read more)

Hypothesis: Unlike the language models before it and ignoring context length issues, GPT-3's primary limitation is that it's output mirrors the distribution it was trained on. Without further intervention, it will write things that are no more coherent than the average person could put together. By conditioning it on output from smart people, GPT-3 can be switched into a mode where it outputs smart text.

According to Gwern, it fails the Parity Task.


I did not believe you so went and checked the internet archive. Sure enough, all the old posts with a ToC are off center. I did not notice until now.

No worries :D This happens to me all the time as well with various parts of our design.

Nitpick, is there a reason why the margins are so large?

The content on the front page is noticeably off center to the right on 1440x900 monitors.

Edit: The content is noticeably off center to the right in general.

The post page centeredness is the way it's always been. Posts are centered when there is no Table of Contents, but with the Table of Contents it looks better (IMO) if it's moved a bit further to the right, to make space for the ToC.
Yeah, noticed that as well. Will fix today. (This was the result of a fix to make it look more centered on larger monitors, and ended up changing the laptop-size-experience by accident)

On the standardization and interoperability side of things. There's been effort to develop decentralized social media platforms and protocols. Most notably being the various platforms of the Fediverse. Together with opensource software, this let's people build large networks that keep the value of network effects while removing monopoly power. I really like the idea of these platforms, but due to the network monopoly of existing social media platforms I think they'll have great difficulty gaining traction.

What's the going rate for audio recordings on Fiverr?

oh yeah, good point. checking the rate for a few for 1615 words. here's my search: [] (note: 1 CAD cost 0.74 USD) * 228 CAD ( []) * 120 CAD ( []) * 78 CAD ( [] ) * 234 CAD ( []) * 9,125 CAD ( [] ) * 50 CAD ( [] ) * 120 CAD ( []) * 313 CAD (

With the ongoing drama that is currently taking place. I'm worried that the rationalist community will find itself inadvertently caught up in the culture war. This might cause a large influx of new users who are more interested in debating politics than anything else on LW.

It might be a good idea to put a temporary moratorium/barriers on new signups to the site in the event that things become particularly heated.

Something in this space seems pretty plausible to me. We are always monitoring contributions from new users, so I think we would notice relatively quickly, and I agree that as soon as we see a suspicious uptick we might want to limit contributions, but I do think I would want to wait until we see the initial signs.
Yup. This is already a thing we keep an eye out for for new users (I'm less likely to approve a new user if they seem primarily interested in arguing politics), and I agree it makes more sense to be on the lookout for it right now.

Organizations, and entire nations for that matter, can absolutely be made to "feel fear". The retaliation just needs to be sufficiently expensive for the organization. Afterwards, it'll factor in the costs of that retaliation when deciding how to act. If the cost is large enough, it won't do things that will trigger retaliation.

Oops, meant to cancel, rather than post. I don't agree, and it's probably not useful to debate. s/for the organization/for many influential members of the organization/ Yes, they _can_ be manipulated and threatened in this way. But not easily, and not without pretty significant commitment on the part of a coordinated and resource-heavy attacker. Below the threshold of "

There is no guarantee that it is learning particularly useful representations just because it predicts pixel-by-pixel well which may be distributed throughout the GPT,

Personally, I felt that that wasn't really surprising either. Remember that this whole deep learning thing started with exactly what OpenAI just did. Train a generative model of the data, and then fine tune it to the relevant task.

However, I'll admit that the fact that theres an optimal layer to tap into, and that they showed that this trick works specifically with transformer autoregressive models is novel to my knowledge.

Being able to accomplish something is important even if it was predicted to be possible. No one is surprised that generative models do embody a lot of useful knowledge (that's much of the point), but it can be hard to tap into it. The difference between GPT & iGPT for transfer learning is that GPT can be queried directly via its modality by putting in text: "Translate this into French", "what genre of text is this?", "tldr", etc. On the other hand, if you were querying iGPT by handing it half an image and expecting it to complete it in a useful way, there is absolutely nothing surprising about that being useful, obviously; but I have a hard time thinking of how you could implement classification by image completion! You normally have to get the knowledge out a different way, through an embedding which can be fed into a linear classification layer; if you can't do that, it's unclear what exactly you do. It was unclear how you use Sparse Transformers, PixelRNN, GANs, etc to do any of that. Now it's clearer. As an analogous example, consider textual style transfer. You can't do it (pre-GPT-3, anyway). Do char-RNNs and Transformers understand the difference between authors and styles and content? Are they capable of textual style transfer? I would be shocked if they weren't. Probably, yes, after all, they can uncannily mimic authors and write plausibly about all sorts of content. But nevertheless, they lack a Gram matrix like CNNs you can easily optimize to do style transfer with. So, no one can do it. Someone finally figuring out how to do it would be big news even if the end output is not surprising.

This isn't news, we knew that sequence predictors could model images for almost a decade now and openAI did the same thing last year with less compute, but no one noticed.

I very definitely noticed Sparse Transformer, but what you're missing is that Sparse Transformers showed good compression performance but was small-scale & primarily about describing the Sparse Transformer/showing it works, and there's nothing about few-shot/transfer learning. There is no guarantee that it is learning particularly useful representations just because it predicts pixel-by-pixel well which may be distributed throughout the GPT, somewhat like the problem in finding the equivalent of Gram matrices in text models (unlike the semi-supervised CNNs where you can expect the embedding or pre-embedding to distill all the knowledge into one place, by design), and you can see in iGPT that getting the representation out is nontrivial - you can easily pick a bad layer to use as the embedding.
5Daniel Kokotajlo3y
Thanks for pointing this out--funnily enough, I actually read the OpenAI thing last year and thought it was cool, but then forgot about it by the time this came out! (The thing from a decade ago I hadn't heard of)

I'll quote myself:

Many of the users on LW have their real names and reputations attached to this website. If LW were to come under this kind of loosely coordinated memetic attack, many people would find themselves harassed and their reputations and careers could easily be put in danger. I don't want to sound overly dramatic, but the entire truth seeking and AI safety project could be hampered by association.

That's why even though I remain anonymous, I think it's best if I refrain from discussing these topics at anything except the meta level on LW. Ev

... (read more)
Ah. Many people on Less Wrong use real names or traceable pseudonyms. If Less Wrong becomes associated with [unspeakable] then anyone who uses [traceable name] on Less Wrong could, by association, be threatened by a mob regardless of whether [traceable name] in particular endorses [unspeakable] because terrorist mobs are not known for their precise discrimination of targets. You illustrate this with a real-world example.
Load More