All of quanticle's Comments + Replies

Why isn’t there a standardized test given by a third party for job relevant skills?

That's what Triplebyte was trying to do for programming jobs. It didn't seem to work out very well for them. Last I heard, they'd been acquired by Karat after running out of funding.

My intuition here is “actually fairly good.” Firms typically spend a decent amount on hiring processes—they run screening tests, conduct interviews, look at CVs, and ask for references. It’s fair to say that companies have a reasonable amount of data collected when they make hiring decisions, and generally, the people involved are incentivized to hire well.

Every part of this is false. Companies don't collect a fair amount of data during the hiring process, and the data they do collect is often irrelevant or biased. How much do you really learn about a c... (read more)

On top of all that, this whole process is totally unaccountable and for some market failure reason, every company repeats it.

Unaccountable : the reason a candidate wasn't hired isn't disclosed, which means in many cases it may be a factually false reason, illegal discrimination, the job wasn't real, or immigration fraud.  Or just "they failed to get lucky on an arbitrary test that measures nothing".

Repeats it: So each company wastes at least a full day of each candidates time, and for each candidate they consider, they waste more than that of their ow... (read more)

Do you know a healthy kid who will do nothing?

Yes. Many. In fact, I'd go so far as to say that most people in this community, who claim that they're self-motivated learners who were stunted by school would have been worse off without the structure of a formal education. One only needs to go through the archives and look at all the posts about akrasia to find evidence of this.

What does "lowercase 'p' political advocacy" mean, in this context? I'm familiar with similar formulations for "democratic" ("lowercase 'd' democratic") to distinguish matters relating to the system of government from the eponymous American political party. I'm also familiar with "lowercase 'c' conservative" to distinguish a reluctance to embrace change over any particular program of traditionalist values. But what does "lowercase 'p' politics" mean? How is it different from "uppercase 'P' Politics"?

Answer by quanticleDec 11, 20232621

A great example of a product actually changing for the worse is Microsoft Office. Up until 2003, Microsoft Office had the standard "File, Edit, ..." menu system that was characteristic of desktop applications in the '90s and early 2000s. For 2007, though, Microsoft radically changed the menu system. They introduced the ribbon. I was in school at the time, and there was a representative from Microsoft who came and gave a presentation on this bold, new UI. He pointed out how, in focus group studies, new users found it easier to discover functionality with th... (read more)

This would be an example of internal vs external validity: it may well be the case in that in their samples of newbies, when posed the specific tasks for the first time, the ribbon worked well and the benefit was statistically established to some very high level of posterior probability; however, just because it was better in that exact setting...

This reminds me of Dan Luu's analysis of Bruce Tog's infamous claim that Apple's thorough user-testing experiments proved, no matter how butthurt it makes people like me, that using the mouse is much faster than ... (read more)

3-year later follow-up: I bought a Hi-Tec C Coleto pen for my brother, who is in a profession where he has to write a lot, and color code forms, etc. He likes it a lot. Thanks for the recommendation.

On the other hand, if plaintiff has already elicited testimony from the engineer to the effect that the conversation happened, could defendant try to imply that it didn’t happen by asking the manager whether he recalled the meeting? I mean, yes, but it’s probably a really bad strategy. Try to think about how you would exploit that as plaintiff: either so many people are mentioning potentially life-threatening risks of your product that you can’t recall them all, in which case the company is negligent, or your memory is so bad it was negligent for you to h

... (read more)
I think we've gone well past the point of productivity on this. I've asked some lawyers for opinions on this. I'll just address a few things briefly. I agree this is true in general, but my point is limited to the cases where documentation would in fact exist were it not for the company's communication policy or data retention policy. If there was a point in the Google case you brought up earlier where Google had attempted to cast doubt on a DOJ witness by pointing out the lack of corroborating evidence (which would have been deleted per Google's policy), I'd strongly reconsider my opinion. What the article about the case said was just that DOJ complained that it would like to have all the documentation that Google destroyed, and that this probably contained evidence which proved their case. It did not say that Google challenged DOJ witnesses on a lack of corroboration between their testimony and the discoverable record. It doesn't have to be the Google case. Any case where the defense tried to impeach a witness on grounds of lack of corroborating evidence where that evidence would have been intentionally destroyed by a data retention policy would do. There are other things I disagree with, but as I said, we're being unproductive.

The thing I said that the defendant would not dispute is the fact that the engineer said something to them, not whether they should have believed him.

I still disagree. If it wasn't written down, it didn't happen, as far as the organization is concerned. The engineer's manager can (and probably will) claim that they didn't recall the conversation, or dispute the wording, or argue that while the engineer may have said something, it wasn't at all apparent that the problem was a serious concern.

There's a reason that whistleblowers focus so hard on generatin... (read more)

So obviously I violently disagree with this, so assuming it was supposed to be meaningful and not some kind of throwaway statement, you should clarify exactly what you do and don't mean by this. They may say this, but I think that you aren't thinking clearly enough in terms of the logical chain of argument that the hypothetical legal proceeding is trying to establish. A lawyer has a witness answer specific questions because they support specific facts which logically prove his case. They don't just say stuff. Suppose plaintiff didn't have a witness, but wanted to try to establish that the company knew about the widgets in order to establish responsibility and/or negligence. Plaintiff might ask a manager "Did any engineers mention that the widgets were dangerous?" And he might reply "I don't recall" at which point plaintiff is SOL. On the other hand, if plaintiff has already elicited testimony from the engineer to the effect that the conversation happened, could defendant try to imply that it didn't happen by asking the manager whether he recalled the meeting? I mean, yes, but it's probably a really bad strategy. Try to think about how you would exploit that as plaintiff: either so many people are mentioning potentially life-threatening risks of your product that you can't recall them all, in which case the company is negligent, or your memory is so bad it was negligent for you to have your regularly-delete-records policy. It's like saying I didn't commit sexual harassment because we would never hire a woman in the first place. Sure, it casts doubt on the opposition's evidence, but at what cost? Disputing the wording is probably a bad idea; arguing that the engineer did say something, but we decided it was not an issue is what I've been saying they would do. But it involves admitting that the conversation happened, which is most or all of what a discoverable record would establish in the first place. Suppose the engineer originally told management that the widget

If you notice something risky, say something. If the thing you predicted happens, point out the fact that you communicated it.

I think this needs to be emphasized more. If a catastrophe happens, corporations often try to pin blame on individual low-level employees while deflecting blame from the broader organization. Having a documented paper trail indicating that you communicated your concerns up the chain of command prevents that same chain from labeling you as a "rogue employee" or "bad apple" who was acting outside the system to further your personal reputation or financial goals.

Plaintiff wants to prove that an engineer told the CEO that the widgets were dangerous. So he introduces testimony from the engineer that the engineer told the CEO that the widgets were dangerous. Defendant does not dispute this.

Why wouldn't the defendant dispute this? In every legal proceeding I've seen, the defendant has always produced witnesses and evidence supporting their analysis. In this case, I would expect the defendant to produce analyses showing that the widgets were expected to be safe, and if they caused harm, it was due to unforeseen circ... (read more)

  You're not reading carefully enough. The thing I said that the defendant would not dispute is the fact that the engineer said something to them, not whether they should have believed him. This is why I said later on Of course the company will defend their decision in either case. My point is about what you gain by having a record of having raised a concern versus testifying that you raised that concern. My opinion is that they're the same unless there's a reason to doubt that you raised the concern like you say you did. And if the defendant doesn't challenge the claim that the concern-raising happened, why would there be? Yes, this is correct. I was simplifying it.  I don't doubt it, but I think you're missing the point here. What I'm referring to by "defendant's strategy" is not the practice of regularly deleting things, but the trial strategy of attempting to rebut witness testimony by claiming that testimony is not corroborated by records while simultaneously regularly deleting records. I agree that regularly deleting things can be very useful to your legal strategy, it just takes certain options off the table for you. Either you can rebut testimony by saying it doesn't match the record or you can regularly delete that record, but you can't do both without getting crucified.

Would it be as self evidently damning as you think it would be? If so, then why would a company like Google explicitly pursue such a weak strategy? It's not just Google either. When I worked at a different FAANG company, I was told in orientation to never use legal terminology in e-mail, for similar reasons.

Google did not pursue that strategy. Or at least, if they did, the article you linked doesn't say so. What I am saying that Google did not and would not do is that when Barton testified that Google would not respond that he was talking horseshit and if what he was saying was true, why isn't there any evidence of it in our internal employee communications? They would not say this because DOJ would say that this corroborating evidence did not exist because Google took steps to ensure it would not. Same here. I was told not to say that this new change would allow us to "Crush Yahoo in terms of search result quality". But I understood the idea to be that since in real life what we were trying to do was just maximize search result quality, we shouldn't let our jocularity and competitive spirit create more work for Legal. Of course, maybe the real FAANG I worked for wasn't Google and I'm just adapting the real story to Google for anonymity purposes. Who knows?

The first lawyer will be hardly able to contain his delight as he asks the court to mark “WidgetCo Safe Communication Guidelines” for evidence.

Having safe communication guidelines isn't as damning as you think it is. The counsel for WidgetCo would merely reply that the safe communication guidelines are there to prevent employees from accidentally creating liabilities by misusing legal language. This is no different than admonishing non-technical employees for misusing technical language.

Indeed this was Google's actual strategy.

It's not that safe communication guidelines are damning. It's that claiming that the lack of discoverable evidence corroborating your statement disproves it while simultaneously having conspired to ensure that discoverable evidence would not exist would be damning.

Games, unlike many real life situations, are entered into by choice. If you are not playing to win, then one must ask why are you bothering to play? Or, more specifically, why are you playing this game and not some other?

3Seth Herd5mo
That's what the whole post was about. You don't seem to be engaging with it, just contradicting it without addressing any of the arguments.

Have you read Playing To Win, by David Sirlin? It makes many of the points that you make here, but it doesn't shy away from winning as the ultimate goal, as you seem to be doing. Sirlin doesn't fall into the trap of lost purposes. He keeps in mind that the goal is to win. Yes, of course, by all means try new strategies and learn the mechanics of the game, but remember that the goal is victory.

4Seth Herd5mo
It's foolish to accept a final goal someone else gives you, let alone a piece of paper in a box. If you're not thinking about why you want to win, you're being foolish. I'm sure Sirlin goes into why winning is a good goal, but you haven't given us any clues here.
2mako yass5mo
It's possible I would have encountered some of this when I used to read game design theory like a decade ago. Here's one where he acknowledges a tradeoff between winning now and winning long term

was militarily weakened severely

That's another highly contentious assertion. Even at the height of Vietnam, the US never considered Southeast Asia to be the main domain of competition against the Soviet Union. The primary focus was always on fielding a military force capable of challenging the Soviets in Western Europe. Indeed, one of the reasons the US failed in Vietnam is because the military was unwilling to commit its best units and commanders to what the generals perceived was a sideshow.

why the US allied with China against the USSR

Was the US e... (read more)

each one after 1900 was followed by either the Cuban Missile Crisis and the US becoming substantially geopolitically weaker than the USSR after losing the infowar over Vietnam

I'm sorry, what? That's a huge assertion. The Vietnam War was a disaster, but I fail to see how it made the US "significantly geopolitically weaker". One has to remember that, at the same time that the US was exiting Vietnam, its main rival, the Soviet Union, was entering a twenty-five year period of economic stagnation that would culminate in its collapse.

I looked into it, this is the kind of research that's really hard to get good info on. I need to do some digging, but generally, it's well known that the US had a historically unprecedented public opinion catastrophe (basically in free fall, by the standards of the time), was militarily weakened severely which was why the US allied with China against the USSR (the USSR asserting military forces on China's border was a costly indicator of Soviet strength and Chinese turmoil), and failing to prevent the oil shocks in formerly US-friendly middle eastern regimes, which were economic catastrophes that each could have done far more damage if luck was worse (if they were mission-critical for the US economy, why couldn't the CIA keep the oil going?). Meanwhile, the USSR remained strong militarily in spite of the economic stagnation. I just found out that some historians might be claiming that the US wasn't really weakened much at all, which absolutely REEKS of the usual suspects. Of course, it's not hard to believe that the US moved much closer to parity with the USSR whereas during the 50s 60s and 70s it was the leader due to being vastly economically superior and substantially technologically superior. But the idea that the US was doing fine after Vietnam, including relative to the Soviets, is not very easy to believe, all things considered.

Chevron deference means that judges defer to federal agencies instead of interpreting the laws themselves where the statute is ambiguous.

Which is as it should be, according to the way the US system of government is set up. The legislative branch makes the law. The executive branch enforces the law. The judicial branch interprets the law. This is a fact that every American citizen ought to know, from their grade-school civics classes.

For example, would you rather the career bureaucrats in the Environmental Protection Agency determine what regulations a

... (read more)

I think they will probably do better and more regulations than if politicians were more directly involved

Why do you think this?

Furthermore, given the long history of government regulation having unintended consequences as a result of companies and private individuals optimizing their actions to take advantage of the regulation, it might be the case that government overregulation makes a catastrophic outcome more likely.

While overturning Chevron deference seems likely to have positive effects for many industries which I think are largely overregulated, it seems like it could be quite bad for AI governance. Assuming that the regulation of AI systems is conducted by members of a federal agency (either a pre-existing one or a new one designed for AI as several politicians have suggested), I expect that the bureaucrats and experts who staff the agency will need a fair amount of autonomy to do their job effectively. This is because the questions relevant AI regulation (i. e.

... (read more)

Why do you think that the same federal bureaucrats who incompetently overregulate other industries will do a better job regulating AI?

Chevron deference means that judges defer to federal agencies instead of interpreting the laws themselves where the statute is ambiguous. It's not so much a question of overregulation vs underregulation as it is about who is doing the interpretation. For example, would you rather the career bureaucrats in the Environmental Protection Agency determine what regulations are appropriate to protect drinking water or random jud... (read more)

2Daniel Kokotajlo7mo
Note that NickGabs doesn't necessarily think that at all. For example, I agree with the quoted paragraph NickGabs wrote, but also I don't expect the same federal bureaucrats who incompetently overregulate other industries to do a better job regulating AI.
Perhaps agencies consistently overregulate. And when it comes to AI, overregulation is preferable to underregulation, whereas for most other fields the opposite is true.

Sometimes if each team does everything within the rules to win then the game becomes less fun to watch and play

Then the solution is to change the rules. Basketball did this. After an infamous game where a team took the lead and then just passed the ball around to deny it to their opponents, basketball added a shot clock, to force teams to try to score (or else give the ball to the other team). (American) Football has all sorts of rules and penalties ("illegal formation", "ineligible receiver downfield", "pass interference", etc) whose sole purpose is to... (read more)

Isn’t this stupid? To have an extra set of ‘rules’ which aren’t really rules and everyone disagrees on what they actually are and you can choose to ignore them and still win the game?

Yes, it is stupid.

Games aren't real life. The purpose of participating in a game is to maximize performance, think laterally, exploit mistakes, and do everything you can, within the explicit rules, to win. Doing that is what makes games fun to play. Watching other people do that, at a level that you could never hope to reach is what makes spectator sports fun to watch.

Imagi... (read more)

I don't know if you read the rest of the piece, but the point I was trying to make is that sometimes this isn't true! Sometimes if each team does everything within the rules to win then the game becomes less fun to watch and play (you may disagree, but many sports fans feel this way). I already gave some examples where this happens in other sports, so I don't see the need for your list of hypotheticals (and I feel like they are strawmen anyway). For what its worth, I agree with you on Bairstow/Carey but which side you take on it is irrelevant (though I can see you are quite passionate about it!).  The piece was about the 'meta' aspects of games which try to address these kind of issues.

This pattern is even more pronounced in motorsport. The history of Formula 1 is the story of teams finding ways to tweak their cars to gain an advantage, other teams whining about unfairness, and the FIA then tweaking the rules to outlaw the "innovation".

Examples include:

... (read more)

any therapeutic intervention that is now standardized and deployed on mass-scale has once not been backed by scientific evidence.

Yes, which is why, in the ideal case, such as with the polio vaccine, we take great effort to gather that evidence before declaring our therapeutic interventions safe and efficacious.

Ah, but how do you make the artificial conscience value aligned with humanity? An "artificial conscience" that is capable of aligning a superhuman AI... would itself be an aligned superhuman AI.

1Oliver Siegel10mo
Correct! That's my point with the main post. I don't see anyone discussing conscience, I mostly hear them contemplate consciousness or computability.  As far as how to actually do this, I've dropped a few ideas on this site, they should be listed on my profile.

We’ve taught AI how to speak, and it appears that openAI has taught their AI how to produce as little offensive content as possible.

The problem is that the AI can (and does) lie. Right now, ChatGPT and its ilk are a less than superhuman levels of intelligence, so we can catch their lies. But when a superhuman AI starts lying to you, how does one correct for that? If a superhuman AI starts veering off in a direction that is unexpected, how does one bring it back on track?

@gwern short story, Clippy highlights many of the issues with naively training a sup... (read more)

1Oliver Siegel10mo
Makes perfect sense! Isn't that exactly why we should develop an artificial conscience, to prevent an AI from lying or having a shadow side?  A built in conscience would let the AI know that lying is not something it should do. Also, using a conscience in the AI algorithm would make the AI combat it's own potential shadow. It'll have knowledge of right and wrong / good or bad, and it's even got superhuman ability to orient itself towards that which is good & right, rather than to be "seduced" by the dark side.

How would you measure the usage? If, for example, Google integrates Bard into its main search engine, as they are rumored to be doing, would that count as usage? If so, I would agree with your assessment.

However, I disagree that this would be a "drastic" impact. A better Google search is nice, but it's not life-changing in a way that would be noticed by someone who isn't deeply aware of and interested in technology. It's not like, e.g. Google Maps navigation suddenly allowing you to find your way around a strange city without having to buy any maps or decipher local road signs.

What I'm questioning is the implicit assumption in your post that AI safety research will inevitably take place in an academic environment, and therefore productivity practices derived from other academic settings will be helpful. Why should this be the case when, over the past few years, most of the AI capabilities research has occurred in corporate research labs?

Some of your suggestions, of course, work equally well in either environment. But not all, and even the ones which do work would require a shift in emphasis. For example, when you say professors ... (read more)

I have 2 separate claims: 1. Any researcher, inside or outside of academia, might consider emulating attributes successful professors have in order to boost personal research productivity.  2. AI safety researchers outside of academia should try harder to make their legible to academics, as a cheap way to get more good researchers thinking about AI safety.  This assumption is not implicit, you're putting together (1) and (2) in a way which I did not intend.  I agree but this is not a counterargument against my post. This is just an incredibly reasonable interpretation of what it means to be "good at networking" for a industry researcher.  My post is not literally recommending that non-academics 80/20 their teaching. I am confused why you think that I would think this. 80/20ing teaching is an example of how professors allocate their time to what's important. Professors are being used as a case study in the post. When applied to an AI safety researcher who works independently or as part of an industry lab, perhaps "teaching" might be replaced with "responding to cold emails" or "supervising an intern". I acknowledge that professors spend more time teaching than non-academic researchers spend doing these tasks. But once again, the point of this post is just to list a bunch of things successful professors do, and then non-professors are meant to consider these points and adapt the advice to their own environment.  This seems like a crux. It seems like I am more optimistic about leveraging academic labor and expertise, and you are more optimistic about deploying AI safety solutions to existing systems.  This is another crux. We both have heard different anecdotal evidence and are weighing it differently.  I never said that academia would take over AI safety research, and I also never said this would be a good thing. I believe that there is a lot of untapped free skilled labor in academia, and AI safety researchers should put in more of an effort (e.g. by writing

What is the purpose, beyond mere symbolism, of hiding this post to logged out users when the relevant data is available, in far more detail, on Google's official AI blog?

2the gears to ascension10mo
just don't want to be the ones helping things like this go viral. I would post more news here if I had a solid sense of who was benefiting from my news-gathering. I'd like to be able to make posts only visible to some specific group; I still wouldn't be posting anything not already public, and my taste is somewhat iffy, but I haven't done more newsposts of this kind than I have for related reasons.
Symbolism is coordination. Not contributing to destroying the world with your own hands, even if you can't stop others from doing it, is a good norm. Iterations of doing concerning things at least a little bit less than others.

I am saying that successful professors are highly successful researchers

Are they? That's why I'm focusing on empirics. How do you know that these people are highly successful researchers? What impressive research findings have they developed, and how did e.g. networking and selling their work enable them to get to these findings? Similarly, with regards to bureaucracy, how did successfully navigating the bureaucracy of academia enable these researchers to improve their work?

The way it stands right now, what you're doing is pointing at some traits that c... (read more)

There are lots of ways a researcher can choose to adopt new productivity habits. They include: 1. Inside view, reasoning from first principles  2. Outside view, copying what successful researchers do The purpose of this post is to, from an outside view perspective, list what a class of researchers (professors) does, which happens to operate very differently from AI safety. Once again, I am not claiming to have an inside view argument in favor of the adoption of each of these attributes. I do not have empirics. I am not claiming to have an airtight causal model. If you will refer back to the original post, you will notice that I was careful to call this a list of attributes coming from anecdotal evidence, and if you will refer back to the AI safety section, you will notice that I was careful to call my points considerations and not conclusions.  You keep arguing against a claim which I've never put forward, which is something like "The bullshit in academia (publish or perish, positive results give better papers) causes better research to happen." Of course I disagree with this claim. There is no need to waste ink arguing against it.  It seems like the actual crux we disagree on is: "How similar are the goals success in academia with success in doing good (AI safety) research?" If I had to guess the source of our disagreement, I might speculate that we've both heard the same stories about the replication crisis, the inefficiencies of grant proposals and peer review, and other bullshit in academia. But, I've additionally encountered a great deal of anecdotal evidence indicating: in spite of all this bullshit, the people at the top seem to overwhelmingly not be bogged down by it, and the first-order factor in them getting where they are was in fact research quality. The way to convince you of this fact might be to repeat the methodology used in Childhoods of exceptional people, but this would be incredibly time consuming. (I'll give you 1/20th of such a blog post

Well, augmenting reality with an extra dimension containing the thing that previously didn’t exist is the same as “trying and seeing what would happen.” It worked swimmingly for the complex numbers.

No it isn't. The difference between and the values returned by is that can be used to prove further theorems and model phenomena, such as alternating current, that would be difficult, if not impossible to model with just the real numbers. Whereas positing the existence of is just like positing the existence of a finite value that s... (read more)

1Thoth Hermes10mo
You can prove additional facts about the world with those values, that was the point of my usage of 'i' as one of the examples.  For h = 1/0 you can upgrade R to the projectively extended real line. If I'm not mistaken, one needs to do this in order to satisfy additional proofs in real analysis (or upgrade to this one instead).  You seem to be asking whether or not doing so in every conceivable case would prove to be useful. I'm saying that we'd be likely to know beforehand about whether or not it would be. Like finding polynomial roots, one might be inclined to wish that all polynomials with coefficients in the reals had roots, therefore, upgrading the space to the complex numbers allows one to get their wish.  

For what it's worth, I had a very similar reaction to yours. Insects and arthropods are a common source of disgust and revulsion, and so comparing anyone to an insect or an arthropod, to me, shows that you're trying to indicate that this person is either disgusting or repulsive.

I'm sorry!  I'm sincerely not trying to indicate that.  Duncan fascinates and unnerves me but he does not revolt me.  I think that "weird bug" made sense to my metaphor generator instead of "weird plant" or "weird bird" or something is that bugs have extremely widely varying danger levels - an unfamiliar bug may have all kinds of surprises in the mobility, chemical weapons, aggressiveness, etc. department, whereas plants reliably don't jump on you and birds are basically all just WYSIWYG; but many weird bugs are completely harmless, and I simply do not know what will happen to me if I poke Duncan.

Probabilities as credences can correspond to confidence in propositions unrelated to future observations, e.g., philosophical beliefs or practically-unobservable facts. You can unambiguously assign probabilities to ‘cosmopsychism’ and ‘Everett’s many-worlds interpretation’ without expecting to ever observe their truth or falsity.

You can, but why would you? Beliefs should pay rent in anticipated experiences. If two beliefs lead to the same anticipated experiences, then there's no particular reason to choose one belief over the other. Assigning probabilit... (read more)

4Eric Chen10mo
Because the meaning of statements does not, in general, consist entirely in observations/anticipated experiences, and it makes sense for people to have various attitudes (centrally, beliefs and desires) towards propositions that refer to unobservable-in-principle things. Accepting that beliefs should pay rent in anticipated experience does not mean accepting that the meaning of sentences are determined entirely by observables/anticipated experiences. We can have that the meanings of sentences are the propositions they express, and the truth-conditions of propositions are generally states-of-affairs-in-the-world and not just observations/anticipated experiences. Eliezer himself puts it nicely here: "The meaning of a statement is not the future experimental predictions that it brings about, nor isomorphic up to those predictions [...] you can have meaningful statements with no experimental consequences, for example:  "Galaxies continue to exist after the expanding universe carries them over the horizon of observation from us."" As to how to choose one belief over another, if both beliefs are observationally equivalent in some sense, there are many such considerations. One is our best theories predict it: if our best cosmological theories predict something does not cease to exist the moment it exits our lightcone, then we should assign higher probability to the statement "objects continue to exist outside our lightcone" than the statement "objects vanish at the boundary of our lightcone". Another is simplicity-based priors: the many-worlds interpretation of quantum mechanics is strictly simpler/has a shorter description length than the Copenhagen interpretation (Many-Worlds = wave function + Schrödinger evolution; Copenhagen interpretation = wave function + Schrödinger evolution + collapse postulate), so we should assign a higher prior to many-worlds than to Copenhagen. If your concern is instead that attitudes towards such propositions have no behavioural implicati

One crude way of doing it is saying that a professor is successful if they are a professor at a top 10-ish university.

But why should that be the case? Academia is hypercompetitive, but the way it selects is not solely on the quality of one's research. Choosing the trendiest fields has a huge impact. Perhaps the professors that are chosen by prestigious universities are the ones that the prestigious universities think are the best at drawing in grant money and getting publications into high-impact journals, such as Nature, or Science.

Specifically I th

... (read more)
  That is not what I am saying. I am saying that successful professors are highly successful researchers, that they share many qualities (most of which by the way have nothing to do with social prestige), and that AI safety researchers might consider emulating these qualities.  This is a non sequitur. I'm not saying stop the blog posts. In fact, I am claiming that "selling your work" is a good thing. Therefore I also think blog posts are fine. When I write about the importance of a good abstract/introduction, I mean not just literally in the context of a NeurIPS paper but also more broadly in the context of motivating ones' work better, so that a broader scientific audience can read your work and want to build off it. (But also separately I do think people should eventually turn good blog posts into papers for wider reach) I disagree. Non-EA funding for safety is pouring in. Safety is being talked about in mainstream venues. Also more academic papers popping up, as linked in my post. In terms of progress on aligning AI I agree the field is in its early stages, but in terms of the size of the field and institutions built up around it, nothing about AI safety feels early stage to me anymore.  I am confused by your repeated focus on empirics, when I have been very up front that this is a qualitative, anecdotal, personal analysis. 

I was thinking more about the inside view/outside view distinction, and while I agree with Dagon's conclusion that probabilities should correspond to expected observations and expected observations only, I do think there is a way to salvage the inside view/outside view distinction. That is to treat someone saying, "My 'inside view' estimate of event is ," as being equivalent to someone saying that . It's a conditional probability, where they're telling you what their probability of a given outcome is, assuming that their understan... (read more)

4Sami Petersen10mo
FWIW I think this is wrong. There's a perfectly coherent framework—subjective expected utility theory (Jeffrey, Joyce, etc)—in which probabilities can correspond to many other things. Probabilities as credences can correspond to confidence in propositions unrelated to future observations, e.g., philosophical beliefs or practically-unobservable facts. You can unambiguously assign probabilities to 'cosmopsychism' and 'Everett's many-worlds interpretation' without expecting to ever observe their truth or falsity. This is reasonable. If a deterministic model has three free parameters, two of which you have specificied, you could just use your prior over the third parameter to create a distribution of model outcomes. This kind of situation should be pretty easy to clarify though, by saying something like "my model predicts event E iff parameter A is above A*" and "my prior P(A>A*) is 50% which implies P(E)=50%." But generically, the distribution is not coming from a model. It just looks like your all things considered credence that A>A*. I'd be hesitant calling a probability based on it your "inside view/model" probability.

I agree that all of these attributes are plausible attributes of successful professors. However, I'd still like to know where you're drawing these observations from? Is it personal observation? And if so, how have you determined whether a professor is successful or not? Is there a study that correlates academic impact across these traits?

  Yes, personal observation, across quite a few US institutions.  One crude way of doing it is saying that a professor is successful if they are a professor at a top 10-ish university. Academia is hypercompetitive so this is a good filter. Additionally my personal observations are skewed toward people who I think do good research, so additionally "successful" here means "does research which electroswing thinks is good".  I haven't looked for one. A lot of them seem tough to measure, hence my qualitative analysis here.  ---------------------------------------- In my experience, successful professors are often significantly better at the skills I've listed than similarly intelligent people who are not successful professors. My internal model is that this is because aptitude in these skills is necessary to survive academia, so anybody who doesn't make the cut never becomes a successful professor in the first place. Specifically I think professors are at least +2σ at "hedgehog-y" and "selling work" compared to similarly intelligent people who are not successful professors, and more like +σ at the other skills.  You can imagine a post "Attributes of successful athletes", where the author knows a bunch of top athletes, and finds shared traits in which the athletes are +2σ or +σ  such as 1) good sleep hygiene, 2) always does warm ups, 3) almost never eats junk food, 4) has a good sports doctor and so on. Even in the absence of a proper causal study, the average person who wants to improve fitness can look at this list and think: "Hmm (4) seems only relevant for professionals, but (1) and (3) seem like they probably have a strong causal effect and (2) seems plausible but hard to tell." 

I agree that the betting approach is better at clarification, but the problem is that it's often too much better. For example, if I say, I'll bet $10 at 80% odds that the weather tomorrow will be sunny, the discussion rapidly devolves into the definitional question of what is a sunny day, exactly? Do I win if I see the sun at any point in the day? Is there a certain amount of cloud cover at which point the day no longer counts as sunny? Where is the cloud cover measured from? If the sky starts out with < 5% clouds, clouds over to > 50%, but then the ... (read more)

Well, I'm not sure how you can have both well-defined propositional probabilities AND undefined, "colloquial" inexact meanings. I think I'd use the word "progresses" rather than "devolves".  This is necessary to clarify what you actually assign 80% chance to happening. You can absolutely do so, but you need to recognize that the uncertainty makes your prediction a lot less valuable to others.  "80% chance that it might conceivably be considered sunny" is just less precise than "80% chance that the weather app at noon will report sunny".   If someone disagrees, and you care about it, you'll need to define what you're disagreeing on.  If nobody cares, then hand-waving is fine.
3Garrett Baker10mo
For sunny days you can just get a reliable reporter to tell you whether its sunny.

We value morality because of evolution. Not because its rational.

Why are those two things mutually exclusive? We understand that is true for the legs of a right triangle, because we have brains that are the result of evolution. Does that make the Pythagorean Theorem "irrational" or untrue, somehow?

This seems to make a jump from “the prompt requires agency to execute well” to “the AI develops the cognitive capability for agency”?

In my scenario the AI already has the cognitive capability for agency. It's just that the capability is latent until the right prompt causes it to be expressed. We've seen early examples of this with ChatGPT, where, if you ask it to plan something or think about adversarial scenarios, it will demonstrate agent-ish behavior.

My point is that while current AIs are probably incapable of having agency, future AIs probably will ... (read more)

I feel like a lot of the objections around agency are answered by the Clippy scenario, and gwern's other essay on the topic, Tool AIs want to be Agent AIs. The AGI need not start with any specific goal or agency. However, the moment it starts executing a prompt that requires it to exhibit agency or goal directed behavior, it will. And at that point, unless the goal is set up such that the agent pursues its goal in a manner that is compatible with the continued existence of humanity over the long term, humanity is doomed. Crafting a goal in this manner is v... (read more)

This seems to make a jump from "the prompt requires agency to execute well" to "the AI develops the cognitive capability for agency"? I read Sarah's point as being that current AIs are fundamentally incapable of having agency (as she defines it). If that's the case, it doesn't matter if the prompt requires the AI to have agency to execute the prompt well: instead, the AI will just fail to execute the prompt well.

I think LLMs are great and plausibly superhuman at language

I think the problem might be that "language" encompasses a much broader variety of tasks than image generation. For example, generating poetry with a particular rhyming structure or meter seems to be a pretty "pure" language task, yet even GPT-4 struggles with it. Meanwhile, diffusion models with a quarter of the parameter count of GPT-4 can output art in a dizzying variety of styles, from Raphael-like neoclassical realism to Picasso-like cubism.

Okay, that's all fair, but it still doesn't answer my question. We don't do any of these things for diffusion models that output images, and yet these diffusion models manage to be much smaller than models that output words, while maintaining an even higher level of output quality. What is it about words that makes the task different?

Or are you suggesting that image generators could also be greatly improved by training minimal models, and then embedding those models within larger networks?

I'm not sure that "even higher level of output quality" is actually true, but I recognize that it can be difficult to judge when an image generation model has succeeded. In particular, I think current image models are fairly bad at specifics in much the same way as early language models.  But I think the real problem is that we seem to still be stuck on "words". When I ask GPT-4 a logic question, and it produces a grammatically correct sentence that answers the logic puzzle correctly, only part of that is related to "words" -- the other part is a nebulous blob of reasoning.  I went all the way back to GPT-1 (117 million parameters) and tested next word prediction -- specifically, I gave a bunch of prompts, and I looked for only if the very next word was what I would have expected. I think it's incredibly good at that! Probably better than most humans.  No, because this is already how image generators work. That's what I said in my first post when I noted the architectural differences between image generators and language models. An image generator, as a system, consists of multiple models. There is a text -> image space, and then an image space -> image. The text -> image space encoder is generally trained first, then it's normally frozen during the training of the image decoder.[1] Meanwhile, the image decoder is trained on a straightforward task: "given this image, predict the noise that was added". In the actual system, that decoder is put into a loop to generate the final result. I'm requoting the relevant section of my first post below: 1. ^ Refer to figure 2 in Or read this: This is the idea that I'm saying could be applied to language models, or rather, to a thing that we want to demonstrate "general intelligence" in the form of reasoning / problem solving / Q&A / planning / etc. First train a LLM, then train a larger system with the LLM as a component within it.

That's a fair criticism, but why would it apply to only language models? We also train visual models with a randomized curriculum, and we seem to get much better results. Why would randomization hurt training efficiency for language generation but not image generation?

First, when we say "language model" and then we talk about the capabilities of that model for "standard question answering and factual recall tasks", I worry that we've accidentally moved the goal posts on what a "language model" is. 

Originally, a language model was a stochastic parrot. They were developed to answer questions like "given these words, what comes next?" or "given this sentence, with this unreadable word, what is the most likely candidate?" or "what are the most common words?"[1] It was not a problem that required deep learning.

Then... (read more)

On the flip side, as gwern pointed out in his Clippy short story, it's possible for a "neutral" GPT-like system to discover agency and deception in its training data and execute upon those prompts without any explicit instruction to do so from its human supervisor. The actions of a tool-AI programmed with a more "obvious" explicit utility function is easier to predict, in some ways, than the actions of something like ChatGPT, where the actions that it's making visible to you may be a subset (and a deliberately deceptively chosen subset) of all the actions that it is actually taking.

It's not that odd. Ars Technica has a good article on why generative AIs have such a strong tendency to confabulate. The short answer is that, given a prompt (consisting of tokens, which are similar to, but not quite the same as words), GPT will come up with new tokens that are more or less likely to come after the given tokens in the prompt. This is subject to a temperature parameter, which dictates how "creative" GPT is allowed to be (i.e. allowing GPT to pick less probable next-tokens with some probability). The output token is added to the prompt, and ... (read more)

Thanks! This was a very helpful comment for me. 

This creates a recursive loop such that each of them experiences what it is like to experience being them experiencing what it is like to be the other, on and on to whatever degree is desired by either of them.

Why should this be the case? When I encounter a potentially hostile piece of programming, I don't run it on my main computer. I run it in a carefully isolated sandbox until I've extracted whatever data or value I need from that program. Then I shut down the sandbox. If the AI is superintelligent enough to scan human minds as its taking humans apar... (read more)

1Thoth Hermes1y
I don't see why it wouldn't be able to do so. I assume that when it does this, it can "pull out" safely.

I’m the idiot holding the hand axe. I’m the imbecile mangling my shins with rock debris. Why bother?

Because no matter how obsolete hand axes or hand-forged iron spoons, or hand-built CPUs are, it's still cool to make one yourself.

True enough. Made me think about this anime from a couple seasons ago, "Do It Yourself!". It's a typical cute girls doing cute things setup (in this case, making furniture and handmade jewelry), but it has a nice thematic undercurrent in how it portrays a world that's more automated than ours, there's casual discussions of AI and Singularity and one character decrying manual labour as obsolete, but the ethos of it all is "actually sometimes working to make stuff is just fun!".

If realism is false, nothing matters, so it’s not bad that everyone dies

That's a misunderstanding of moral realism. Moral realism is a philosophical argument that states that moral arguments state true facts about the world. In other words, when I say that "Murder is bad," that is a fact about the world, as true as or the Pythagorean theorem.

It's entirely possible for me to think that moral realism is false (i.e. morality is a condition of human minds) while also holding, as a member of humanity, a view that the mass extinction of all humanity is ... (read more)

I like this way of putting it. In Principia Mathematica, Whitehead and Russell spent over 300 pages laying groundwork before they even attempt to prove 1+1=2. Among other things, they needed to define numbers (especially the numbers 1 and 2), equality, and addition. I do think "1+1=2" is an obvious fact. If someone claimed to be intelligent and also said that 1+1=3, I'd look at them funny and press for clarification. Given all the assumptions about how numbers work I've absorbed over the course of my life, I'd find it hard to conceive of anything else. Likewise, I find it hard to conceive of any alternative to "murder is bad," because over the course of my life I've absorbed a lot of assumptions about the value of sentient life. But the fact that I've absorbed these assumptions doesn't mean every intelligent entity would agree with them. In this analogy, the assumptions underpinning human morality are like Euclid's postulates. They seem so obvious that you might just take them for granted, as the only possible self-consistent system. But we could have missed something, and one of them might not be the only option, and there might be other self-consistent geometries/moralities out there. (The difference being that in the former case M.C. Escher uses it to make cool art, and in the latter case an alien or AI does something we consider evil.)
I don't think it's a misunderstanding of moral realism.  I think that versions of moral anti-realism don't capture things really mattering, for reasons I explain in the linked post.  I also don't think rocks have morality--the idea of something having morality seems confused.  

This would have a significant impact on the everyday lives of people within a month.

It would have a drastic impact on your life in a month. However, you are a member of a tiny fraction of humanity, sufficiently interested and knowledgeable about AI to browse and post on a forum that's devoted to a particularly arcane branch of AI research (i.e. AI safety). You are in no way representative. Neither am I. Nor is, to a first approximation, anyone who posts here.

The average American (who, in turn, isn't exactly representative of the world) has only a vague ... (read more)

I'm a bit confused where you're getting your impression of the average person / American, but I'd be happy to bet on LLMs that are at least as capable as GPT3.5 being used (directly or indirectly) on at least a monthly basis by the majority of Americans within the next year?
FWIW South Park released an episode this season with ChatGPT as its main focus. I do think that public perception is probably moving fairly quickly on things like this these days. But I agree with you generally that if you're posting here about these things then you're likely more on the forefront than the average American.

That very same argument could have been (and was!) made for steam engines, electric motors, and (non-AI) computers. In 2023, we can look back and say, "Of course, as workers were displaced out of agriculture and handicrafts, they went into manufacturing and services." But it wasn't apparent in 1823 that such a transition would occur. Indeed, in 1848, one of the motivations for Marx to write The Communist Manifesto was his fervent belief that industrial manufacturing and services would not be able to absorb all the surplus workers, leading to significant alienation and, eventually, revolution.

Load More