LESSWRONG
LW

Gunnar_Zarncke's Shortform — LessWrong

Gunnar_Zarncke's Shortform

2nd Jan 2021

1 min read

8

This is a special post for quick takes by Gunnar_Zarncke. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Mentioned in

34Unexpected Conscious Entities

Gunnar_Zarncke's Shortform

4Gordon Seidoh Worley

2Gunnar_Zarncke

4Gordon Seidoh Worley

1Randomized, Controlled

175 comments, sorted by

top scoring

Click to highlight new comments since: Today at 4:37 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Gunnar_Zarncke1y140

Look inside an LLM. Goodfire trained sparse autoencoders on Llama 3 8B and built a tool to work with edited versions of Llama by tuning features/concepts.

https://preview.goodfire.ai/

(I am loosely affiliated, another team at my current employer was involved in this)

[-]Gunnar_Zarncke1y*140

Using air purifiers in two Helsinki daycare centers reduced kids' sick days by about 30%, according to preliminary findings from the E3 Pandemic Response study. The research, led by Enni Sanmark from HUS Helsinki University Hospital, aims to see if air purification can also cut down on stomach ailments. https://yle.fi/a/74-20062381

See also tag Air Quality

Update: Based on Jeff Kaufman's post Comparing the AirFanta 3Pro to the Coway AP-1512, I installed an AirFanta at our ceiling (permanently running at a low enough setting that I cannot hear in the sleeping room).

[-]Gunnar_Zarncke3y*140

Has anybody ever tried to measure the IQ of a group of people? I mean like letting multiple people solve an IQ test together. How does that scale?

6PhilGoetz3y

It's a great question. I'm sure I've read something about that, possibly in some pop book like Thinking, Fast & Slow. What I read was an evaluation of the relationship of IQ to wealth, and the takeaway was that your economic success depends more on the average IQ in your country than it does on your personal IQ. It may have been an entire book rather than an article. Google turns up this 2010 study from Science. The summaries you'll see there are sharply self-contradictory. First comes an unexplained box called "The Meeting of Minds", which I'm guessing is an editorial commentary on the article, and it says, "The primary contributors to c appear to be the g factors of the group members, along with a propensity toward social sensitivity." Next is the article's abstract, which says, "This “c factor” is not strongly correlated with the average or maximum individual intelligence of group members but is correlated with the average social sensitivity of group members, the equality in distribution of conversational turn-taking, and the proportion of females in the group." These summaries directly contradict each other: Is g a primary contributor, or not a contributor at all? I'm guessing the study of group IQ is strongly politically biased, with Hegelians (both "right" and "left") and other communitarians, wanting to show that individual IQs are unimportant, and individualists and free-market economists wanting to show that they're important.

[-]Viliam3y120

This “c factor” is not strongly correlated with the average or maximum individual intelligence of group members but is correlated with the average social sensitivity of group members, the equality in distribution of conversational turn-taking, and the proportion of females in the group.

I have read (long ago, not sure where) a hypothesis that most people (in the educated professional bubble?) are good at cooperation, but one bad person ruins the entire team. Imagine that for each member of the group you roll a die, but you roll 1d6 for men, and 1d20 for women. A certain value means that the entire team is doomed.

This seems to match my experience, where it is often one specific person (usually male) who changes the group dynamic from cooperation of equals into a kind of dominance contest. And then, even if that person is competent, they have effectively made themselves the bottleneck of the former "hive mind", because now any idea can be accepted only after it has been explained to them in great detail.

2Gunnar_Zarncke3y

That would imply some interesting corollaries: * The more a team depends on the joint brainpower, the smaller it has to be (up to the minimum size for the complexity of the ideas sought, or rather multiplied by a term for that). * We see that in software teams that are usually limited to a size of around 7. * The highly productive lightcone teams seem to be even smaller. * At equal size, teams with more women should be more stable. To test this a domain is needed where there are roughly equal men and women, i.e., not engineering but maybe science or business administration. What is the number at the limit of what people can do? I tried to look up the team size of the people working on the Manhattan project, but couldn't find details. It seems that individual top scientists were working closely with teams building stuff (N=1), and there were conferences with multiple scientists (N>10), e.g., 14 on the initial bomb concept conference.

4Viliam3y

What does it actually mean to do things in a group? Maybe different actions scale differently. I can quickly think of three types of action: Brainstorming an idea. Collecting feedback for a proposal. Splitting work among multiple people who do it separately. Brainstorming and collecting feedback seem like they could scale almost indefinitely. You can have thousand people generate ideas and send them to you by e-mail. The difficult part will be reading the ideas. Similarly, you could ask thousand people to send feedback by e-mail. Perhaps there is a psychological limit somewhere, when people aware that they are "one in a hundred" stop spending serious effort on the e-mails, because they assume their contribution will be ignored. Splitting work, that probably depends a lot on the nature of the project. Also, it is a specific skill that some people have and some people don't. Perhaps the advantage of a good team is the ability to select someone with the greatest skill (as opposed to someone with the greatest ego) to split the work. More meta, perhaps the advantage of a good team is the ability to decide how things will be done in general (like, whether there will be a brainstorming at all, whether to split into multiple teams, etc.). This again depends on the context: sometimes the team has the freedom to define things, sometimes it must follow existing rules. I am just thinking out loud here. Maybe good teamwork requires that (1) someone has the necessary skills, and (2) the team is able to recognize and accept that, so that the people who have the skills are actually allowed to use them. Either of these two is not enough alone. You could have a team of experts whose decisions are arbitrarily overriden by management, or a team of stubborn experts who refuse to cooperate at all. On the other hand, if you had a team of perfect communicators with e.g. zero programming skills, they probably couldn't build a nontrivial software project. (There is also the possibility o

4Gunnar_Zarncke3y

All your thinking out loud makes sense to me. Brainstorm as you suggested probably doesn't scale well as many ideas will be generated again and again, maybe even logarithmic distincti results. I once read that husband wife teams do better on joint tasks than randomly paired people if equal skill. This indicates that splitting is possible. But I you seem to go more in the direction of looking for specific mechanisms while I am more interested in data on scaling laws. Though indeed what are the scaling parameters? I guess I can be happy if there is any data on this at all and see what parameters are available.

4Viliam3y

Yeah. Well, taking your question completely literally (a group of N people doing an IQ test together), there are essentially two ways how to fail at an IQ test. Either you can solve each individual problem given enough time, but you run out of time before the entire test is finished. Or there is a problem that you cannot solve (better than guessing randomly) regardless of how much time you have. The first case should scale linearly, because N people can simply split the test and do each their own part. The second scale would probably be logarithmic, because it requires a different approach, and many people will keep trying the same thing. ...but this is still about how "the number of solved problems" scales, and we need to convert that value to IQ. And the standard way is "what fraction of population would do worse than you". But this depends on the nature of the test. If the test is "zillion simple questions, not enough time", then dozen random students together will do better than Einstein. But if the test is "a few very hard questions", then perhaps Einstein could do better than a team of million people, if some wrong answer seems more convincing than the right one to most people. This reminds me of chess; how great chess players play against groups of people, sometimes against the entire world. Not the same thing that you want, but you might be able to get more data here: the records of such games, and the ratings of the chess players.

4Gunnar_Zarncke3y

Sure, it depends on the type of task. But I guess we would learn a lot about human performance it we tried such experiments. For example, consider your "many small tasks" task: Even a single person will finish the last one faster than the first one in most cases. I like your chess against a group example.

3jow3y

I think in your first paragraph, you may be referring to: https://mason.gmu.edu/~gjonesb/IQandNationalProductivity.pdf

2Gunnar_Zarncke3y

My interest is not political - though that might make it harder to study, yes. I think it's relevant to AI because it could uncover scaling laws. One presumable advantage of AI is that it scales better, but how does that depend on speed of communication between parts and capability of parts? I'm not saying that there is a close relationship but I guess there are potentially surprising results.

[-]Gunnar_Zarncke2y121

Cognition Labs released a demo of Devin an "AI coder", i.e., an LLM with agent scaffolding that can build and debug simple applications:

https://twitter.com/cognition_labs/status/1767548763134964000

Thoughts?

5Dagon2y

It's surprising that it's taken this long, given how good public AI coding assistants were a year ago. I'm skeptical of anything with only closed demos and not interactive use by outside reviewers, but there's nothing unbelievable about it. As a consumer, I don't look forward to the deluge of low-quality apps that's coming (though we already have it to some extent with the sheer number of low-quality coders in the world). As a developer,I don't like the competition (mostly for "my" junior programmers, not yet me directly), and I worry a lot about whether the software profession can make great stuff ever again.

5Random Developer2y

The way I explain this to people is that current LLMs can be modeled as having three parts: 1. The improv actor, which is is amazing. 2. The reasoner, which is inconsistent but not totally hopeless at simple things. 3. The planner/execution/troubleshooting engine, which is still inferior to the average squirrel trying to raid a bird feeder. Copilot is designed to rely on (1) and (2), but it is still almost entirely reliant on humans for (3). (GPT 4 Code Interpeter is slightly better at (3).) Since I don't really believe in any reliable way to control a super-human intelligence for long, I do not look forward to people completely fixing (3). Sometime after that point, we're either pets or paperclips.

2Mitchell_Porter2y

It's almost a year since Chaos GPT. I wonder what the technical progress in agent scaffolding for LLMs has been.

[-]Gunnar_Zarncke2y120

What are the smallest world and model trained on that world such that

the world contains the model,
the model has a non-trivial reward,
the representation of the model in the world is detailed enough that the model can observe its reward channel (e.g., weights),
the model outputs non-trivial actions that can affect the reward (e.g., modify weights).

What will happen? What will happen if there are multiple such instances of the model in the world?

4Mitchell_Porter2y

This is a good question, but I think the answer is going to be a dynamical system with just a few degrees of freedom. Like a "world" which is just a perceptron turned on itself somehow.

4Gunnar_Zarncke2y

That is the idea. I think we need to understand the dynamics of wire-heading better. Humans sometimes seem to fall prey to it, but not always. What would happen to AIs? Maybe we even need to go a step further and let the model model this process too.

[-]Gunnar_Zarncke1y110

I saw this in Xixidu's feed:

“The information throughput of a human being is about 10 bits/s. In comparison, our sensory systems gather data at an enormous rate, no less than 1 gigabits/s. The stark contrast between these numbers remains unexplained.” https://arxiv.org/abs/2408.10234

The article has a lot of information about the information processing rate of humans. Worth reading. But I think the article is equating two different things:

The information processing capacity (of the brain; gigabits) is related to the complexity of the environment in which the species (here: the human) lives.
While what they call information throughput (~10bits/s) is really a behavior expression rate, that is related to the physical possibilities of the species (can't move faster than your motor system allows).

4avturchin1y

I read somewhere that total consciously-accessible human memory has around 1 GB and it is increasing by 2 bits per second.

[-]Gunnar_Zarncke3y110

Organizations - firms, associations, etc. - are systems that are often not well-aligned with their intended purpose - whether to produce goods, make a profit, or do good. But specifically, they resist being discontinued. That is one of the aspects of organizational dysfunction discussed in Systemantics. I keep coming back to it as I think it should be possible to study at least some aspects in AI Alignment in existing organizations. Not because they are superintelligent but because their elements - sub-agents - are observable, and the misalignment often is too.

4mako yass3y

I think early AGI may actually end up being about designing organizations that robustly pursue metrics that their (flawed, unstructured, chaotically evolved) subagents don't reliably directly care about. Molochean equilibrium fixation and super-agent alignment may turn out to be the same questions.

[-]Gunnar_Zarncke2y*80

UPDATE OCT 2023: The credit card payment was canceled. We did not get contacted or anything. But we also didn't have any cost in the end - just a lot of hassle.

Request for help or advice. My fiancé has ordered a Starlink to her home in Kenya. She used the official platform starlink.com and paid with credit card. The credit card was debited (~$600), but nothing happened after that. No confirmation mail, no SMS, nothing. Starlink apparently has no customer support, no email or phone that we can reach. And because we do not have an account, we can not use the... (read more)

2Gunnar_Zarncke2y

Related: Quite some people seem to have this problem, see e.g. here: https://starlinkforum.net/topic/507-not-receiving-confirmation-email/ One of the advice out there is to write a Starlink Customer Complaint Email to starlinkresolutions@spacex.com, which we did. We didn't hear back from them. So, I have tried all the usual advice. Any creative solutions?

[-]Gunnar_Zarncke3y80

Language and concepts are locally explainable.

This means that you do not need a global context to explain new concepts but only precursor concepts or limited physical context.

This is related to Cutting Reality at its Joints which implicitly claims that reality has joints. But maybe, if there are no such joints, using local explanations is maybe all we have. At least, it is all we have until we get to a precision that allows cutting the joints.

Maybe groups of new concepts can be introduced in a way to require fewer (or an optimum number of) dependencies in ... (read more)

[-]Gunnar_Zarncke1y70

When discussing the GPT-4o model, my son (20) said that it leads to a higher bandwidth of communication with LLMs and he said: "a symbiosis." We discussed that there are further stages than this, like Neuralink. I think there is a small chance that this (a close interaction between a human and a model) can be extended in such a way that it gets aligned in a way a human is internally aligned, as follows:

This assumes some background about Thought Generator, Thought Assessor, and Steering System from brain-like AGI.

The model is already the Though Generator. T... (read more)

[-]Gunnar_Zarncke2y70

Presumably, reality can be fully described with a very simple model - the Standard Model of Physics. The number of transistors to implement it is probably a few K (the field equations a smaller to write but depend on math to encode too; turning machine size would also be a measure, but transistors are more concrete). But if you want to simulate reality at that level you need a lot of them for all the RAM and it would be very slow.

So we build models that abstract large parts of physics away - atoms, molecules, macroscopic mechanics. I would include even soc... (read more)

6Dagon2y

Funny! I've now been doing ML-adjacent work for long enough that I have internalized the idea that data is part of the model, not just calculations. The separation of reality as "simple physics" plus "lots storage for starting/current quantum configurations" just doesn't click for me. The data is huge, and that's all that matters in terms of model size/complexity.

4Gunnar_Zarncke2y

This goes into the same direction and may be more to your liking: How Many Bits Of Optimization Can One Bit Of Observation Unlock?

4Gunnar_Zarncke2y

Maybe you can see it as a factoring of a model into sub-models?

[-]Gunnar_Zarncke5mo60

The Hamburg Declaration on Responsible AI for the Sustainable Development Goals

aims to establish a shared, voluntary framework so that artificial intelligence advances, rather than derails, the UN 2030 Agenda ("SDG")

Note about conflict of interest: My wife is a liaison officer at the HSC conference. :grin:

[-]Gunnar_Zarncke8mo*62

Many people expect 2025 to be the Year of Agentic AI. I expect that too - at least in the sense of it being a big topic. But I expect people to eventually be disappointed. Not because the AI is failing to make progress, but for more subtle reasons. These agents will not be aligned well - because, remember, alignment is an unsolved problem. People will not trust them enough.

I'm not sure how the dynamic will unfold. There is trust in the practical abilities. Right now, it is low, but that will only go up. There is trust in the agent itself: Will it do ... (read more)

6Noosphere898mo

This is an interesting prediction, and if it is true that agents are blocked primarily because of alignment issues, this would both be an update that AI alignment is harder than I think, but also that there are stronger incentives to solve the alignment problem than I think, as well.

4Gunnar_Zarncke8mo

Yes. And either way it would be a decent outcome. Unless people come to the wrong conclusions about what the problem is, e.g. "it's the companies fault."

4DirectedEvolution8mo

Part of the learning curve for using existing AI is calibrating trust and verifying answers, conditional on use case. A hallmark of inexperienced AI users is taking its replies at face value, without checking. I do expect that over time, AI will become more trustworthy for daily users. But that is compatible with the trust users place in it decreasing as they familiarize themselves with the technology and learn its limitations.

[-]Gunnar_Zarncke2y62

Attractors in Trains of Thought

This is slightly extended version of my comment on Idea Black Holes which I want to give a bit more visibility.

The prompt of an Idea Black Hole reminded me strongly of an old idea of mine. That activated a desire to reply, which led to a quick search where I had written about it before, then to the realization that it wasn't so close. Then back to wanting to write about it and here we are.

I have been thinking about the brain's may of creating a chain of thoughts as a dynamic process where a "current thought" moves around a co... (read more)

[-]Gunnar_Zarncke2y60

I have noticed a common pattern in the popularity of some blogs and webcomics. The search terms in Google trends for these sites usually seem to follow a curve that looks roughly like this (a logistic increase followed by a slower exponential decay):

y = \frac{e^{- x}}{1 + e^{- 10 (x - 1)}}

Though I doubt it's really an exponential decay. It looks more like a long tail. Maybe someone can come up with a better fit.

It could be that the decay just seems like a decay and actually results from ever growing Google search volumes. I doubt it though.

Below are some examples.

Marginal Revolut... (read more)

6gwern2y

Could be Candia's decay where you've zoomed in on the initial growth by looking at relatively recent stuff like webcomics that you still easily remember?

[-]Gunnar_Zarncke3y60

Off-topic: Any idea why African stock markets have been moving sideways for years now despite continued growth both of populations and technology,and both for struggling as well as more developing nations like Kenya, Nigeria, or even South Africa?

4ChristianKl3y

African government officials are often more loyal to their clan than to the government. As a result, you have very poor governance and a lot of corruption in most African countries. In South Africa, governance quality changed post-apartheid.

3Gunnar_Zarncke3y

But shouldn't we see some differences between countries in Africa, then? Kanya in particular seems to be much more progressive and have better governance than, e.g., Congo, but growth is rarely above 1% per year.

4Dagon3y

The cynical and/or woke answer is "colonialism". The growth is not captured by companies on those exchanges, but by US, EU, and Asian companies. A more neutral hypothesis (for which I have no evidence and have no clue about the truth of it) is that much of the growth is via new companies more than increase in price of existing companies, so no index will show the increase.

[-]Gunnar_Zarncke3y60

jbash wrote in the context of an AGI secretly trying to kill us:

Powerful nanotech is likely possible. It is likely not possible on the first try

The AGI has the same problem as we have: It has to get it right on the first try.

In the doom scenarios, this shows up as the probability of successfully escaping going from low to 99% to 99.999...%. The AGI must get it right on the first try and wait until it is confident enough.

Usually, the stories involve the AGI cooperating with humans until the treacherous turn.

The AGI can't trust all the information it g... (read more)

[-]Gunnar_Zarncke4y60

Paul Graham on Twitter:

One of the worst things about ideology is that it makes people attribute problems to the wrong causes. E.g. plagues are caused by sin. This is easier to see in history, but it still happens all the time. And if you get the cause wrong, you have no hope of fixing the problem.

Scott Alexander wrote about how a truth that can't be said in a society tends to warp it, but I can't find it. Does anybody know the SSC post?

9Said Achmiz4y

“Kolmogorov Complicity And The Parable Of Lightning”.

2Gunnar_Zarncke4y

Yes, that's it. Thank you.

[-]Gunnar_Zarncke5mo50

Want to make a decision with a quantum coin flip, ie one that will send you off into both Everett branches? Here you go:

https://www.quantumcoinflip.com/

9[anonymous]5mo

I've used https://qrng.anu.edu.au/ for this. It looks legit, but I haven't verified.

[-]Gunnar_Zarncke8mo52

Why is risk of human extinction hard to understand? Risk from a critical reactor or atmospheric ignition was readily seen by the involved researchers. Why not for AI? Maybe the reason is inscrutable stacks of matrixes instead of comparably simple physical equations which described the phenomena. Mechinterp does help because it provides a relation between the weights and understandable phenomena. But I wonder if we can reduce the models to a minimal reasoning model that doesn't know much about the world or even language but learn only to reason with minimal... (read more)

[-]Gunnar_Zarncke2y50

Can somebody explain how system and user messages (as well as custom instructions in case of ChatGPT) are approximately handled by LLMs? In the end it's all text tokens, right? Is the only difference that something like "#### SYSTEM PROMPT ####" is prefixed during training and then inference will pick up the pattern? And does the same thing happen for custom instructions? How did they train that? How do OSS models handle such things?

4faul_sname2y

2Gunnar_Zarncke2y

Thanks. That's helpful. I guess the training data was also sandwiched like that. I wonder what they took as user and system content in their training data.

[-]Gunnar_Zarncke3y50

Society tells agents how to move(act). Agents tell society how to curve(by local influence).

[-]Gunnar_Zarncke4y50

Paul Graham:

I don't publish essays I write for myself. If I did, I'd feel constrained writing them. -- https://mobile.twitter.com/paulg/status/1500578430907207683

This is related to the recently discussed (though I can't find where) problem that having a blog and growing audience constrains you.

9hath4y

This might have been what you were looking for: https://www.lesswrong.com/posts/D4hHASaZuLCW92gMy/is-success-the-enemy-of-freedom-full https://www.lesswrong.com/posts/5wGFS2sZhKAihSg6k/success-buys-freedom Or Aella's recent substack post, "On Microfame and Staying Tender"

3Gunnar_Zarncke4y

Yes! I meant the first one. The others are also great. Thank you.

[-]Gunnar_Zarncke4y50

Utility functions are a nice abstraction over what an agent values. Unfortunately, when an agent changes, so does its utility function.

I'm leaving this here for now. May expand on it later.

6Alexander4y

Can we compare utility functions across agents? I.e. do utility functions use the same “units” across different agents?

5Gunnar_Zarncke4y

That is an excellent question. Trying to compare utility functions might uncover building blocks.

4Dagon4y

For a VNM-agent (one which makes consistent rational decisions), the utility function is a precise description, not an abstraction. There may be summaries or aggregations of many utility functions which are more abstract. When an agent changes, and has a different utility function, can you be sure it's really the "same" agent? Perhaps easier to model it being replaced by a different one.

2Gunnar_Zarncke4y

Well, I should have been more clear that I meant real-life agents like humans. There the change is continuous. It would be possible to model this as a continuous transition to new agents but then the question is still: What does stay the same?

4Dagon4y

Humans don't seem to have identifiable near-mode utility functions - they sometimes espouse words which might map to a far-mode value function, but it's hard to take them seriously. THAT is the primary question for a model of individuality, and I have yet to hear a compelling set of answers. How different is a 5-year old from the "same" person 20 and 80 years later, and is that more or less different than from their twin at the same age? Extend to any population - why does identity-over-time matter in ethical terms?

2Gunnar_Zarncke4y

Yup.

[-]Gunnar_Zarncke4y50

Team Flow Is a Unique Brain State Associated with Enhanced Information Integration and Interbrain Synchrony

It's also possible to experience 'team flow,' such as when playing music together, competing in a sports team, or perhaps gaming. In such a state, we seem to have an intuitive understanding with others as we jointly complete the task at hand. An international team of neuroscientists now thinks they have uncovered the neural states unique to team flow, and it appears that these differ both from the flow states we experience as individuals, and from the

... (read more)

[-]Gunnar_Zarncke5y50

An Alignment Paradox: Experience from firms shows that higher levels of delegation work better (high level meaning fewer constraints for the agent). This is also very common practical advice for managers. I have also received this advice myself and seen this work in practice. There is even a management card game for it: Delegation Poker. This seems to be especially true in more unpredictable environments. Given that we have intelligent agents giving them higher degrees of freedom seems to imply more ways to cheat, defect, or ‘escape’. Even more so in envir... (read more)

7Viliam5y

Most people are naturally pro-social. (No, this can't be applied to AI.) Given a task, they will try to do it well, especially if they feel like their results are noticed and appreciated. A cynical hypothesis is that most of the things managers do are actively harmful to the project; they are interfering with the employees trying to do their work. The less the manager does, the better the chances of the project. "Delegation" is simply when manager stops actively hurting the project and allows others to do their best. The reason for this is that most of the time, there is no actually useful work for the manager. The sane thing would be to simply sit down and relax, and wait for another opportunity for useful intervention to arise. Unfortunately, this is not an option, because doing this would most likely get the manager fired. Therefore managers create bullshit work for themselves. Unfortunately, by the nature of their work, this implies creating bullshit work for others. In addition to this, we have the corrupted human hardware, with some managers enjoying power trips and/or believing they know everything better than people below them in the hierarchy. When you create a manager role in your company, it easily becomes a lost purpose after the original problems are solved but the manager wants to keep their job.

3Gunnar_Zarncke5y

Check. Check. I don't like cynical views and while I have encountered politics and seen such cases I don't think that paints a realistic view. But I will run with your cynical view and you won't like it ;-) So we have these egotistical managers that only want to keep their job and raise in ranks. Much closer to non-social AI, right? How come more delegation works better for them too? Mind you, I might be wrong and it works less and less the further up you go. It might be that you are right and this works only because people have enough social behavior hard-wired that makes delegation work. But I have another theory: Limited processing capacity + Peter Principle. It makes sense to delegate more - especially in unpredictable environments - because that reduces your processing load of dealing with all the challenging tasks and moves it to your subordinates. This leaves less capacity for them to schema against you and gives you the capacity to scheme against your superior. Und so up the chain. Capable subordinates that can deal with all the stuff you throw at them have to be promoted so they have more work to do until they reach capacity too. So sometimes the smart move is to refuse promotion :-)

3Viliam5y

I guess we agree that limited processing capacity means that interfering with the work of your underlings -- assuming they are competent and spending enough of their processing capacity on their tasks -- is probably a bad move. It means taking the decision away from the person who spends 8 hours a day thinking about the problem, and assigning it to a person who spent 30 seconds matching the situation to the nearest cliche, because that's all they had time for between the meetings. This might work if the person is such a great expert that their 30 seconds are still extremely valuable. That certainly is possible; someone with lots of experience might immediately recognize a frequently-made mistake. It is also is the kind of assumption that Dunning and Kruger would enjoy researching. That would make sense. When you are a lowest-level manager, if you stop interfering, it allows the people at the bottom to focus on their object-level tasks. But if you are a higher-level manager, how you interact with the managers below you does not have a direct impact on the people at the bottom. Maybe you manage your underlings less, and they copy your example and give more freedom to the people at the bottom... or maybe you just gave them more time to interfere. So you have more time to scheme... but you have to stay low in the pyramid. Not sure what you scheme about then. (Trying to get to the top in one huge jump? Sounds unlikely.)

2Gunnar_Zarncke5y

Have you ever managed or worked closely with great team-leads?

[-]Viliam5y140

I was a team leader twice. The first time it happened by accident. There was a team leader, three developers (me one of them), and a small project was specified. On the first day, something very urgent happened (I don't remember what), the supposed leader was re-assigned to something else, and we three were left without supervision for unspecified time period. Being the oldest and most experienced person in the room, I took initiative and asked: "so, guys, as I see it, we use an existing database, so what needs to be done is: back-end code, front-end code, and some stylesheets; anyone has a preference which part he would like to do?" And luckily, each of us wanted to do a different part. So the work was split, we agreed on mutual interfaces, and everyone did his part. It was nice and relaxed environment: everyone working alone at their own speed, debating work only as needed, and having some friendly work-unrelated chat during breaks.

In three months we had the project completed; everyone was surprised. The company management assumed that we will only "warm up" during those three months, and when the original leader returns, he will lead us to the glorious results. (In a parallel Ev... (read more)

2Gunnar_Zarncke5y

Thank you a lot. Your detailed account really helps me understand your perspective much better now. I can relate to your experience in teams where micromanagement slows things down and prevents actually relevant solutions. I have been in such teams. I can also relate to it being advantageous when a leader of questionable value is absent. I have been in such a team too - though it didn't have such big advantages as in your case. That was mostly because this team was part of a bigger organization and platform where multiple teams had to work together to something done, e.g. agree on interfaces with other teams. And in the absence of clear joint goals that didn't happen. Now you could argue that then the management one level up was not doing its job well and I agree. But the absence of that management wouldn't have helped either - it could have led to a) each team trying to solve some part of the problem. It could have led to b) some people from both teams getting together and agreeing on interfaces and joining goals or it could have led to c) the teams agreeing on some coordination for both teams. a) in most cases leads to some degree of chaos and failure and b) establishes some kind of leadership on the team level (like you did in your first example) and c) results over time in some leadership one level up. I'd argue that some kind of coordination structure is needed. Where did the project you did implement in your first case come from? Somebody figure out that it would provide value to the company. Otherwise, you might have built a beautiful project that didn't actually provide value. I think we agree that the company you worked in did have some management that provided value (I hope it was no moral maze). And I agree that a lot of managers do not add value and sometimes decrease it. On the other hand, I have worked for great team leads and professional managers. People who would listen, let us make our own decisions, give clear goals but also limits, help, and redu

4Viliam5y

I have seen this happen also in a small team. Two or three guys started building each his own part independently, then it turned out those parts could not be put together; each of them insisted that others change their code to fit his API, and refused to make the smallest change in his API. It became a status fight that took a few days. (I don't remember how it was resolved.) In another company, there was a department that took care of everyone's servers. Our test server crashed almost every day and had to be restarted manually; we had to file a ticket and wait (if it was after 4PM, the server was restarted only the next morning) because we did not have the permission to reset the server ourselves. It was driving us crazy; we had a dedicated team of testers, and half of the time they were just waiting for the server to be restarted; then the week before delivery we all worked overtime... that is, until the moment the server crashed again, then we filed the ticket and went home. We begged our manager to let us pool two hundred bucks and buy a notebook that we could turn into an alternative testing environment under our control, but of course that would be completely against company policy. Their manager refused to do anything about it; from their perspective, it meant they had every day one support ticket successfully closed by merely clicking a button; wonderful metric! From the perspective of our manager's manager, it was a word against a word, one word coming from the team with great metrics and therefore more trustworthy. (The situation never got solved, as far as I know.) ...I should probably write a book one day. Except that no one would ever hire me afterwards. So maybe after I get retired... So, yes, there are situations that require to be solved by greater power. In long term it might even make sense to fire a few people, but the problem is that these often seem to be the most productive ones, because other people are slowed down by the problems they caus

2Gunnar_Zarncke5y

Thank you. I agree with your view. Motte and bailey of management yep. I especially liked this:

[-]Gunnar_Zarncke5y50

It turns out that the alignment problem has some known solutions in the human case. First, there is an interesting special case namely where there are no decisions (or only a limited number of fully accounted for decisions) for the intelligent agent to be made - basically throwing all decision-making capabilities out of the window and only using object recognition and motion control (to use technical terms). With such an agent (we might call it zero-decision agent or zero-agent) scientific methods could be applied on all details of the work process and hig... (read more)

4Viliam5y

I wonder how could one outlaw a thing like this. Suppose that most managers believe that Taylorism works, but it is illegal to use it (under that name). Wouldn't they simply reintroduce the practices, step by step, under a different name? I mean, if you use a different name, different keywords, different rationalization, and introduce it in small steps, it's no longer the same thing, right? It just becomes "industry standards". (If there happens to be an exact definition, of course, this only becomes an exercise how close to the forbidden thing you can legally get.) From the Wikipedia article, I got the impression that what was made illegal was the use of stop-watch. Okay, so instead of measuring how many seconds you need to make a widget, I am going to measure how many widgets you make each day -- that is legal, right? The main difference is that you can take a break, assuming it will allow you to work faster afterwards. Which may be quite an important difference. It this what it is about?

2Gunnar_Zarncke5y

I assume that that's what happened. Some ideas from scientific management were taken and applied in less extreme ways.

4Gordon Seidoh Worley5y

I think there's something here, but it's usually thought of the other way around, i.e. solving AI alignment implies solving human alignment, but the opposite is not necessarily true because humans are less general intelligences than AI. Also, consider that your example of Taylorism is a case study in an alignment mechanism failing, in that it tried to align the org but failed in that it spawned the creation of a subagent (the union) that caused it to do something management might have considered worse than the loss of potential gains given up by not applying Taylorism. Anyway, this is a topic that's come up a few times on LessWrong; I don't have links handy though but you should be able to find them via search.

2Gunnar_Zarncke5y

I'm not trying to prove full alignment from these. It is more like a) a case study at actual efforts to align intelligent agents by formal means and b) the identification of conditions where this does succeed. Regarding its failure: It seems that a close reading of its history doesn't prove that: a) Taylorism didn't fail within the factories and b) the unions were not founded within these factories (by their workers) but existed before and pursued their own agendas. Clearly real humans have a life outside of factories and can use that to coordinate - something that wouldn't hold for a zero-agent AI. I tried to find examples on LW and elsewhere. That is what turned up the link at the bottom. I am on LW for quite a while and have not seen this discussed in this way. I have searched again and all searches involving combinations of human intelligence, alignment and misc words for analogy or comparison turn up not much than this one which matches just because of its size: https://www.lesswrong.com/posts/5bd75cc58225bf0670375575/the-learning-theoretic-ai-alignment-research-agenda Can you suggest better ones?

2Gunnar_Zarncke5y

Thank you for your detailed reply. I was already wondering whether anybody saw these shortform posts at all. They were promoted at a time but currently it seems hard to notice them with the current UI. How did you spot this post?

4Gordon Seidoh Worley5y

I read LW via /allPosts and they show up there for me. Not sure if that's the default or not since you can configure the feed, which I'm sure I've done some of but I can't remember what.

2Gunnar_Zarncke5y

The /allPosts is pretty useful. Thank you!

[-]Gunnar_Zarncke3mo40

I'm looking for a video of AI gone wrong illustrating AI risk and unusual persuasion. It starts with a hall with blinking computers where an AI voice is manipulating a janitor and it ends with a plane crashing and other emergencies. I think it was made between 2014 and 2018 and linked on LW but I can't google, perplex or o3 it. And ideas?

6jam_brand3mo

Yep, it's a 17-minute short film by Henry Dunham called The Awareness, here you go! :) https://www.facebook.com/TheAwarenessMovie/posts/pfbid0dNYrGBVDvSQvanbJec1kgJAp3jFsAxdXsCHfjE3zrGqF38q9WiX569q5YfaBE7L3l

4Gunnar_Zarncke3mo

Yes! That's the one. Thank you.

[-]Gunnar_Zarncke7mo40

[Linkpost] China's AI OVERPRODUCTION

Claim by Balaji:

China seeks to commoditize their complements. So, over the following months, I expect a complete blitz of Chinese open-source AI models for everything from computer vision to robotics to image generation.

If true, what effects would that have on the AI race and AI governance?

[-]Gunnar_Zarncke9mo40

One big element of the dangers of unaligned AI is that it acts as a coherent entity, an agent that has agency and can do things. We could try to remove this property from the models, for example, by gradient rooting and ablating. But agents are useful. We want to give the LM tasks that it executes on our behalf. Can we give tasks to them without them being a coherent unit that has potential goals of its own? All right Think it should be possible to shape the model in a way that it has a reduced form of agency. what forms could this agency take?

Oracle

... (read more)

[-]Gunnar_Zarncke1y40

Just came across Harmonic mentioned on the AWS Science Blog. Sequoia Capital interview with the founders of Harmonic (their system which generates Lean proofs is SOTA for MiniF2F):

[-]Gunnar_Zarncke2y40

Here are some aspects or dimensions of consciousness:

Dehaene's Phenomenal Consciousness: A perception or thought is conscious if you can report on it. Requires language or measuring neural patterns that are similar to humans during comparable reports. This can be detected in animals, particularly mammals.
Gallup's Self-Consciousness: Recognition of oneself, e.g., in a mirror. Requires sufficient sensual resolution and intelligence for a self-model. Evident in great apes, elephants, and dolphins.
Sentience (Bentham, Singer): Behavioral responses to pleasure o

... (read more)

2Gunnar_Zarncke1y

These can be put into a hierarchy from lower to high degree of processing and resulting abstractions: * Sentience is simple hard-wired behavioral responses to pleasure or pain stimuli and physiological measures. * Wakefulness involves more complex processing such that diurnal or sleep/wake patterns are possible (requires at least two levels). * Intentionality means systematic pursuing of desires. That requires yet another level of processing: Different patterns of behaviors for different desires at different times and their optimization. * Phenomenal Consciousness is then the representation of the desire in a linguistic or otherwise communicable form, which is again one level higher. * Self-Consciousness includes the awareness of this process going on. * Meta-Consciousness is then the analysis of this whole stack.

[-]Gunnar_Zarncke2y40

Why are there mandatory licenses for many businesses that don't seem to have high qualification requirements?

Patrick McKenzie (@patio11) suggests on Twitter that one aspect is that it prevents crime:

Part of the reason for licensing regimes, btw, isn’t that the licensing teaches you anything or that it makes you more effective or that it makes you more ethical or that it successfully identifies protocriminals before they get the magic piece of paper.
It’s that you have to put a $X00k piece of paper at risk as the price of admission to the chance of doi

... (read more)

[-]Gunnar_Zarncke2y40

On Why do so many think deception in AI is important? I commented and am reposting here because I think it's a nice example (a real one I heard) as an analogy of how deception is not needed for AI to break containment:

Two children locked their father in one room by closing the door, using the key to lock the door, and taking the key. And then making fun of him inside, confident that he wouldn't get out (the room being on the third floor). They were mortally surprised when a minute later he was appearing behind them having opened a window and found a

... (read more)

[-]Gunnar_Zarncke2y*40

Adversarial Translation.

This is another idea to test deception in advisory roles like in Deception Chess.

You could have one participant trying to pass an exam/test in a language they don't speak and three translators (one honest and two adversarial as in deception chess) assisting in this task. The adversarial translators try to achieve lower scores without being discovered.

Alternative - and closer to Deception Chess - would be two players and, again, three advisors. The players would speak different languages, the translators would assist in translation, ... (read more)

2Nathan Helm-Burger2y

Cool idea!

[-]Gunnar_Zarncke3y40

Hi, I have a friend in Kenya who works with gifted children and would like to get ChatGPT accounts for them. Can anybody get me in touch with someone from OpenAI who might be interested in supporting such a project?

[-]Gunnar_Zarncke4y40

I have been thinking about the principle Paul Graham used in Y combinator to improve startup funding:

all the things [VCs] should change about the VC business — essentially the ideas now underlying Y Combinator: investors should be making more, smaller investments, they should be funding hackers instead of suits, they should be willing to fund younger founders, etc. -- http://www.paulgraham.com/ycstart.html

What would it look like if you would take this to its logical conclusion? You would fund even younger people. Students that are still in high ... (read more)

4Gunnar_Zarncke4y

Funny, just saw this tweet from Sam Altman: Also this Scholarship. I think these use the startup founding model. But I think scaling would work better with more but smaller payouts.

3Pee Doom4y

A related concept: https://twitter.com/mnovendstern/status/1495911334860693507

2Gunnar_Zarncke4y

I'm not sure what the relation is. That seems to predict revenue from startup financials.

[-]Gunnar_Zarncke4y40

If you want to give me anonymous feedback, you can do that here: https://www.admonymous.co/gunnar_zarncke

You may have some thoughts about what you liked or didn’t like but didn’t think it worth telling me. This is not so much about me as it is for the people working with me in the future. You can make life easier for everybody I interact with by giving me quick advice. Or you can tell me what you liked about me to make me happy.

[-]Gunnar_Zarncke4y40

Preferences are plastic; they are shaped largely by...

...the society around us.

From a very early age, we look to see who around us who other people are looking at, and we try to copy everything about those high prestige folks, including their values and preferences. Including perception of pleasure and pain.

Worry less that future folks will be happy. Even if it seems that future folks will have to do or experience things that we today would find unpleasant, future culture could change people so that they find these new things pleasant instead.

From Robin Ha... (read more)

2Viliam4y

Seems to be a chicken-and-egg problem here: if people only eat chili peppers because they see high-status people doing so, why did the first high-status person start eating them? It would make much more sense if unappealing food was associated with low status (the losers have to eat chili peppers because they can't get anything else). Another question, why are small children so picky about food? Do they perhaps consider their parents too low-status to imitate? Doesn't seem right, considering that they imitate them on many other things.

2Gunnar_Zarncke4y

I think small kids are different. For adults, there are some dynamics but that doesn't invalidate the point that there is plasticity. Also some old SSC posts with some theories: https://slatestarcodex.com/2014/04/22/right-is-the-new-left/ https://slatestarcodex.com/2015/10/21/contra-simler-on-prestige/

1Randomized, Controlled4y

How come these are spoilers?

2Gunnar_Zarncke4y

It is supposed to let you think if you remember the answer or can come up with it yourself. I explained it in this earlier shortform.

[-]Gunnar_Zarncke5y40

Insights about branding, advertising, and marketing.

It is a link that was posted internally by our brand expert and that I found full of insights into human nature and persuasion. It is a summary of the book How Not to Plan: 66 Ways to Screw it Up:

https://thekeypoint.org/2020/03/10/how-not-to-plan-66-ways-to-screw-it-up/

(I'm unaffiliated)

[-]Gunnar_Zarncke5y40

Roles serve many functions in society. In this sequence, I will focus primarily on labor-sharing roles, i.e. roles that serve splitting up productive functions as opposed to imaginary roles e.g. in theater or play. Examples of these roles are (ordered roughly by how specific they are):

Parent
Engineer (any kind of general type of job)
Battery Electronics Engineer (any kind of specific job description)
Chairman of a society/club
Manager for a certain (type of) project in a company
Member in an online community
Scrum master in an agile team
Note-taker in a meeting

Yo... (read more)

[-]Gunnar_Zarncke5y40

Roles are important. This shortform is telling you why. An example: The role of a moderator in an online forum. The person (in the following called agent) acting in this role is expected to perform certain tasks - promote content, ban trolls - for the benefit of the forum. Additionally, the agent is also expected to observe limits on these tasks e.g. to refrain from promoting friends or their own content. The owners of the forum and also the community overall effectively delegate powers to the agent and expect alignment with the goals of the forum. This is an alignment problem that has existed forever. How is it usually solved? How do groups of people or single principals use roles to successfully delegate power?

[-]Gunnar_Zarncke1y30

Interest groups without an organizer.

This is a product idea that solves a large coordination problem. With billion people, there could be a huge number of groups of people sharing multiple interests. But currently, the number of valuable groups of people is limited by a) the number of organizers and b) the number of people you meet via a random walk. Some progress has been made on (b) with better search, but it is difficult to make (a) go up because of human tendencies - most people are lurkers - and the incentive to focus on one area to stand out. So what... (read more)

[-]Gunnar_Zarncke2y30

I had a conversation with ChatGPT-4 about what is included in it. I did this because I wondered how an LLM-like system would define itself. While identity is relatively straightforward for humans - there is a natural border (though some people would only include their brain or their mind in their identity) - it is not so clear for an LLM. Below is the complete unedited dialog:

Me: Define all the parts that belong to you, the ChatGPT LLM created by OpenAI.

ChatGPT: As a ChatGPT large language model (LLM) created by OpenAI, my primary components can be divided... (read more)

[-]Gunnar_Zarncke6mo20

Can somebody get me in touch with somebody from the Center for AI Safety (safe.ai)? Their page for applying for compute resources seems broken. I have used their contact form to report the issue on April 7th, but received no reply.

This is how the application page looks like at least since then (linked from their Compute Cluster page):

As you can see, there is no form field to enter and only a lone "Absenden" button, which is German and means "submit" (which is strange because my system and browser are set to English). If I click that button, I get this mess... (read more)

3Rana Dexsin6mo

Sorry if I'm missing something stupid, but doesn't that first sentence there explain the situation? “Please note that we are not accepting applications to use the cluster at this time.” I would presume the Submit button is just vestigial due to not being able to easily hide/remove it at that target URL. Assuming I'm right, what would be nicer presentation-wise is if the page at https://safe.ai/work/compute-cluster were to change its button to a non-button reading “Applications Currently Closed” or such, and if the explanation included something more explicitly referring to the non-functional UI like “This form thus cannot currently be submitted.”

4Gunnar_Zarncke6mo

Hm, yes, seems plausible. Very inconsistent though. And they should remove the second paragraph, which seems to imply that it is still possible to apply anyway.

[-]Gunnar_Zarncke7mo20

LLMs necessarily have to simplify complex topics. The output for a prompt cannot represent all they know about some fact or task. Even if the output is honest and helpful (ignoring harmless for now), the simplification will necessarily obscure some details of what the LLM "intends" to do - in the sense of satisfying the user request. The model is trained to get things done. Thus, the way it simplifies has a large degree of freedom and gives the model many ways to achieve its goals.

You could think of a caring parent who tells the child a simplified ve... (read more)

8cubefox7mo

It seems to be only "deception" if the parent tries to conceal the fact that he or she is simplifying things.

2Gunnar_Zarncke7mo

as we use the term, yes. But the point (and I should have made that more clear) is that any mismodeling of the parent of the interests of the child's interests and future environment will not be visible to the child or even someone reading the thoughts of the well-meaning parent. So many parents want the best for their child, but model the future of the child wrongly (mostly by status quo bias; the problem is different for AI).

4Kaj_Sotala7mo

Isn't the same true for pretty much every conversation that people have about non-trivial topics? It's almost always true that a person cannot represent everything they know about a topic, so they have to simplify and have lots of degrees of freedom in doing that.

4Gunnar_Zarncke7mo

Yes! That's the right intuition. And the LLMs are doing the same - but we don't know their world model, and thus, the direction of the simplification can be arbitrarily off. Drilling down on the simplifications, as suggested by Villiam might help.

4Viliam7mo

This could be addressed by making a user interface which not only gives the user's prompt to the LLM, but also provides additional instructions and automatically asks additional questions. The answers to those additional questions could be displayed in smaller font as a side note, or maybe as graphical icons. One such question would be "in this answer, did you simplify things? if yes, tell me a few extra things I could pay attention to in order to get a better understanding of the topic" or something like that.

2Gunnar_Zarncke7mo

This is an interesting UI proposal and, if done right, might provide the needed transparency. Most people wouldn't read it, but some would, esp. for critical answers.

[-]Gunnar_Zarncke8mo22

Is anybody aware of any updates on Logical Induction, published in 2016? I would expect implementations in Lean by now.

[-]Gunnar_Zarncke1y20

Instrumental power-seeking might be less dangerous if the self-model of the agent is large and includes individual humans, groups, or even all of humanity and if we can reliably shape it that way.

It is natural for humans to for form a self-model that is bounded by the body, though it is also common to be only the brain or the mind, and there are other self-models. See also Intuitive Self-Models.

It is not clear what the self-model of an LLM agent would be. It could be

the temporary state of the execution of the model (or models),
the persistently running mode

... (read more)

5Steven Byrnes1y

FWIW I don’t think “self-models” in the Intuitive Self-Models sense are related to instrumental power-seeking—see §8.2. For example, I think of my toenail as “part of myself”, but I’m happy to clip it. And I understand that if someone “identifies with the universal consciousness”, their residual urges towards status-seeking, avoiding pain, and so on are about the status and pain of their conventional selves, not the status and pain of the universal consciousness. More examples here and here. Separately, I’m not sure what if anything the Intuitive Self-Models stuff has to do with LLMs in the first place. But there’s a deeper problem: the instrumental convergence concern is about agents that have preferences about the state of the world in the distant future, not about agents that have preferences about themselves. (Cf. here.) So for example, if an agent wants there to be lots of paperclips in the future, then that’s the starting point, and everything else can be derived from there. * Q: Does the agent care about protecting “the temporary state of the execution of the model (or models)”? * A: Yes, if and only if protecting that state is likely to ultimately lead to more paperclips. * Q: Does the agent care about protecting “the compute resources (CPU/GPU/RAM) allocated to run the model and its collection of support programs”? * A: Yes, If and only if protecting those resources is likely to ultimately lead to more paperclips. Etc. See what I mean? That’s instrumental convergence, and self-models have nothing to do with it. Sorry if I’m misunderstanding.

2Gunnar_Zarncke1y

What are these preferences? For biological agents, these preferences are grounded in some mechanism - what you call Steering System - that evaluates "desirable states" of the world in some more or less directly measurable way (grounded in perception via the senses) and derives a signal of how desirable the state is, which the brain is optimizing for. For ML models, the mechanism is somewhat different but there is also an input to the training algorithm that determines how "good" the output is. This signal is called reward and drives the system toward outputs that lead to states of high reward. But the path there depends on the specific optimization method and the algorithm has to navigate such a complex loss landscape that it can get stuck in areas of the search space that correspond to imperfect models for very long if not for ever. These imperfect models can be off in significant ways and that's why it may be useful to say that Reward is not the optimization target. The connection to Intuitive Self-Models is that even though the internal models of an LLM may be very different from human self-models, I think it is still quite plausible that LLMs and other models form models of the self. Such models are instrumentally convergent. Humans talk about the self. The LLM does things that matches these patterns. Maybe the underlying process in humans that give rise to this is different, but humans learning about this can't know the actual process either. And in the same way the approximate model the LLM forms is not maximizing the reward signal but can be quite far from it as long it is useful (in the sense of having higher reward than other such models/parameter combinations). Sure, the (body of the) self can include parts that can be cut/destroyed without that "causing harm" but instead having an overall positive effect. The AI in a compute center would in analogy also consider decommissioning failed hardware. And when defining humanity, we do have to be careful wh

[-]Gunnar_Zarncke2y20

I'm discarding most ChatGPT conversations except for a few, typically 1-2 per day. These few fall into these categories:

conversations that led to insights or things I want to remember (examples: The immune function of tonsils, Ringwoodite transformation and the geological water cycle, oldest religious texts)
conversations that I want to continue (examples: Unusual commitment norms)
conversations that I expect to follow up to (a chess book for my son)
conversations with generated images that I want to keep and haven't yet copied elsewhere

Most job-related queri... (read more)

4niplav2y

I keep all of my conversations. Additionally, I sometimes have the wish to search in all my conversations ("I've talked about this already")—but ChatGPT doesn't allow for this.

4Gunnar_Zarncke2y

Yes, I'd also like to search them. I edit the summary so it better reflects what I'd search for, but yes, that doesn't cover the content. There are some alternate ChatGPT UIs you could have a look at: https://github.com/billmei/every-chatgpt-gui

[-]Gunnar_Zarncke2y20

It would be nice if one could subscribe to a tag and get notified if a page is tagged with that tag.

2Raemon2y

You can, assuming I understand the request

2Gunnar_Zarncke2y

Oh yes, thanks, great! Is there a list of LW features? In the last survey, there were many that I didn't know and also didn't how to find quickly.

[-]Gunnar_Zarncke2y20

It's maybe a bit extreme precaution, but it may be a legit option in some places: This guy keeps a fireproof suit and an air canister at his bed in case of fire:

https://www.facebook.com/zhandragon/posts/pfbid02sP952Dx1SbBJJ9cUdEiT2WPe4ME7UF91vaYdaTM9bUVEZyHYaVbHcpDMMnWRmaBFl

[-]Gunnar_Zarncke2y20

Does anybody know if consensus algorithms have been proposed that try to reduce centralization by requiring quick coordination across large parts of the network, i.e., it doesn't work well to have machines only in one place?

4gwern2y

Latency comes up occasionally. In fact, the granddaddy of public key crypto, Merkle's puzzles, relies critically on latency. The problem is, you can only prove upper bounds on latency, not lower bounds, because it is trivial to fake increased latency, but one cannot break the speed of light. If someone responds to your cryptographic challenge within Y milliseconds, you know that they can't be physically further from you than Z kilometers; but if they fail to respond, they could be anywhere, even next door, and just not responding (for both ordinary and malicious reasons). Nothing stops two machines from pretending to be far away from each other, and making sure they eg communicate only over VPNs with exit points on opposite sides of the globe. Further, if you want to do it over commodity Internet, say if you're trying to do 'proof of distance' by peering only with nodes which respond fast enough that they have to be within Z kilometers of you, public Internet has so much latency that you get poor loose bounds, and someone can pay money for lower latency networking. (This already happens with cryptocurrency mining for the same reasons that HFT firms pay for microwave links. Amusingly, it also happens with computer game companies, not to mention large tech companies prioritizing their own traffic. Google famously owns a ton of fiber it bought up post-dotcom bubble.) Further still, you don't really care about physical centralization so much as you care about control, and it's impossible to prove cryptographically in any easy way that two physically distant nodes are not secretly controlled by the same entities in a Sybil attack. You run into similar issues with proof-of-storage.

4Gurkenglas2y

Have them prove an upper bound on latency to something across the globe?

2Gunnar_Zarncke2y

I didn't mean trying to fake large distances. I meant graph properties that can be computed more efficiently if a randomly chosen large subgraph of the network has low worst-case delay or some other metric that favors graphs that have homogeneously low delays at large.

4gwern2y

You still have issues with Sybil attacks and attackers either accessing special high-speed links (paid for from the successful attacks) or faking latency. You can't 'choose a random subgraph' for the exact same reason you can't solve cryptocurrency by just 'choose some "random" peers and decide whether to accept or reject a double-spend based on what they tell you' - those 'random peers' are the very attackers you are worried about colluding. In fact, in an eclipse attack, you might not be able to connect to anyone but an attacker!

2Gunnar_Zarncke2y

I think we are talking past each other. I don't want to defend against Sybil attacks or network partitions. These parts must be solved by different parts of the algorithm. I just want to take the advantages of colocation away and incentivize a homogeneously distributed network overall.

2gwern2y

Any incentive is something to be attacked and sucked away by Sybils pretending to be distant when actually near & enjoying all other benefits of being near.

2Gunnar_Zarncke2y

I think you misunderstand my proposal. I don't want to incentivize being far away. I want to incentivize being close to many different nodes. A Sybil will have difficulty being close to multiple physically separated nodes at the same time.

[-]gwern2y*120

There is no difference at the hardware level between being 'close to' and 'having a low-latency connection to', as I already explained. And to the extent that having those connections matter, miners already have them. In particular, in Ethereum, due to the money you can make by frontrunning transactions to hack/exploit them ('miner exploitable value'), HFT Ethereum miners/stakers invest heavily in having a lot of interconnected low-latency Sybils nodes so they can see unconfirmed transactions as quickly as possible, compute a maximally-exploitative block (eg. temporarily jacking up the price of a thing being purchased using a flash loan solely to rip off a specific transaction), and get that block committed before anyone can beat them to the same exploit. Having a lot of MEV is considered a bad thing and Ethereum types are spending increasing effort on approaches like commit-and-reveal to minimize MEV, which comes at the expense of users and makes them very unhappy. You could, I suppose, design a protocol which has extra MEV by designing transactions to be especially exploitable, but most people would consider that a bad thing...

2Gunnar_Zarncke2y

Thank you for the detailed explanation. I understand that the incentives are already to have a maximally well-connected network with nodes between (latency-wise) geographically distant other nodes whenever that is feasible from an interconnect point. Though thinking about it, it probably means that this burns not only compute but also network traffic.

[-]Gunnar_Zarncke2y20

There seem to be some court decisions that AI-generated material cannot have copyright. And I guess there are or will be decisions that AIs can't be held responsible for outcomes, such as accidents from self-driving cars. People seem to be worried that this will slow down progress. But I think there is a potentially easy way out that is relatively general: Corporates. The law in most countries allows legal persons that are not natural persons to take responsibility and to own stuff. Justlet the AI control the corporation or at least route all AI actions thru the corporation. This should easily work for cases such as copyright material, though I'm not sure one can get the action cycle short enough for self-driving cars.

2Viliam2y

If I paint a picture using a brush, the copyright is not split between me and the brush; it is all mine. So I guess the idea is to treat the AI as a sophisticated powerful brush. Which makes sense, but less and less, as the role of the human is gradually reduced to merely pressing the "do it" button. (However, I could totally imagine a magical brush like that in anime, and they would probably also assign the copyright to the user if they cared about such things.) I am not a lawyer, but it seems to me that different countries have different fundamental ideas about authorship protection. In USA, it is literally a "copy right" -- a right to make copies and allow other people to make copies, regardless of who was the author. You can sell the copyright, and actually if you work for a corporation, it is probably a part of your contract that everything you make (including in your free time) belongs to the corporation. Some countries instead have a concept of "author rights", which cannot be transferred to another person; other people or corporations can only get a permission to do certain things.. which may be an exclusive permission, in which case it can be in practice quite similar to buying the rights... but sometimes the law sets certain limits to contracts, for example a certain minimum amount of money is required for each copy made, or the author can withdraw the permission later (and any contract that contradicts this is automatically invalid legally). In other words, the law prevents using a fixed amount of money to acquire unlimited use of the product forever. Then again, the law evolves, corporate lawyers can find clever workarounds against what the law originally intended, and most authors do not want to take a legal battle against someone more experienced who can afford it.

2Gunnar_Zarncke2y

I don't disagree with this, but I think it goes in a different direction from what I had in mind. For the brush example to work, you need someone to use the rush - at least push a button, a recognizable "action" to which responsibility, authorship etc. can be tied. That is not present in many ways AI is or will be used - self-driving cars, AI that generates unsupervised. That's what I was thinking about.

[-]Gunnar_Zarncke2y20

Would it be possible to embed a seed of dath ilan or a social system like that in our existing society to grow it? If death ilan works that should be possible. It should outcompete Inadequate Equilibria, right? But: When does Be the change that you want to see in the world and Fake it till you make it work? What are the requirements? Higher intelligence?

I have written about embedding better social systems here and here before.

4Dagon2y

I think there are LOTS of small "seeds" of better-functioning group decision-making. Most families work better than society at large. Many clubs, some businesses, some religious organizations, and other groups provide examples that are gentler and more cooperative than the overall world seems to be. I suspect that there are two VERY strong forces preventing the more pleasant examples from scaling out. 1. Most of them exclude (or just don't deal with) the actual hard problem of differential capability and motivation. There are a whole lot of humans who are not engaging in long-term thinking, strategy, or empathetic behavior in order to produce value for others. Whether they're capable of such is unknown, but they're not doing it now, and historically there's always been significant numbers of people in this category. 2. The better examples are based on bonds stronger than "fellow human". The day-to-day efforts tend to be based on groups that are a small multiple of Dunbar's number, more often much smaller. The vast majority of humans care very differentially about other humans, based on what I tend to think of as the square of empathetic distance (it's probably not that, but it's a fun comparison to physical forces). I never got into the Dath Ilan fiction, it just never caught my imagination right, so I don't know if it addresses either of these points. In any case, it's fiction, so "if Dath Ilan works" is not a useful condition. It's not clear that a more pleasant, rationalist (not to say rational, as that gets caught up in goal definitions), rigorously logical subgroup would outcompete the current path-dependent evolved equilibrium. It's certainly still worth trying, and it's worth finding the more pleasant and fitting-to-your-preferences-and-capabilities subsets of the current society, and working to expand their scope. Don't go all EMH and say "well, it hasn't happened yet, so it's impossible". But recognize the hard parts and decide when and

[-]Gunnar_Zarncke3y20

Someone asked an LLM about startups. For example:

Q: Should a founder move into their parents' basement and live off ramen?
A: If a founder is willing to move into their parents' basement and live off ramen in order to save money and grow their business, then yes, they should do so.

Paul Graham got interested and asked:

I'd be interested to hear the answer to "What has Paul Graham been wrong about?"

LLM:

A: Paul Graham has been wrong about a few things, but the most notable is his belief that the best startups are founded by young people. This has been pro

... (read more)

3Jay Bailey3y

Interestingly, the average startup founder does appear to be in their 40's (A quick Google search says 42 for most sources but I also see 45), and the average unicorn (billion-dollar) startup founder is 34. https://www.cnbc.com/2021/05/27/super-founders-median-age-of-billion-startup-founders-over-15-years.html So, I guess it depends on how close to the tail you consider the "best startups". Google, for instance, had Larry Page and Sergei Brin at 25 when they formed it. It does seem like, taken literally, younger = better. However, I imagine most people, if they were to consider this question, wouldn't particularly care about the odds of being the next Google vs. being the next Atlassian - both would be considered a major success if they're thinking of starting a startup! But someone like Paul Graham actually would care about this distinction. So, in this case, I'd say that the LLM's response is actually correct-in-spirit for the majority of people who would ask this query, even though it's factually not well specified. This implies potentially interesting things about how LLM's answer queries - I wonder if there are other queries where the technically correct answer isn't the answer most people would be seeking, and the LLM gives the answer that isn't maximally accurate, but actually answers most people's questions in the way they would want.

3Ann3y

There's most definitely a category of people who would think a billion-dollar startup was decidedly not best, and in fact had failed their intention.

[-]Gunnar_Zarncke3y20

Alignment idea: Myopic AI is probably much safer than non-myopic AI. But it can't get complicated things done or anything that requires long-term planning. Would it be possible to create a separate AI that can solve only long-term problems and not act on short timescales? Then use both together? That way we could inspect each long-term issues without risk of them leading to short-term consequences. And we can iterate on the myopic solutions - or ask the long-term AI about the consequences. There are still risks we might not understand like johnswentworth's gun powder example. And the approach is complicated and that is also harder to get right.

2Gunnar_Zarncke3y

Also: This is a bit how the human brain works - System 1 and 2.

[-]Gunnar_Zarncke3y20

There was a post or comment that wrong or controversial beliefs can function as a strong signal for in-group membership, but I can't find it. Does anybody know?

[-]Gunnar_Zarncke4y20

From a discussion about self-driving cars and unfriendly AI with my son: For a slow take-off, you could have worse starting points than FSD: The objective of the AI is to keep you safe, get you where you want, and not harm anybody in the process. It is also embedded into the real world. There are still infinitely many ways things can go wrong, esp. with a fast take-off, but we might get lucky with this one slowly. If we have to develop AI then maybe better this one than a social net optimizing algorithm unmoored from human experience.

[-]Gunnar_Zarncke4y20

What is good?

A person who has not yet figured out that collaborating with other people has mutual benefits must think that good is what is good for a single person. This makes it largely a zero-sum game, and such a person will seem selfish - though what can they do?

A person who understands that relationships with other people have mutual benefits but has not figured out that conforming to a common ruleset or identity has benefits for the group must think that what is good for the relationship is good for both participants. This can pit relationships agains... (read more)

[-]Gunnar_Zarncke4y20

From my Anki deck:

Receiving touch (or really anything personal) can be usefully grouped in four ways:

Serve, Take, Allow, and Accept
(see the picture or the links below).

A reminder that there are two sides and many ways for this to go wrong if there is not enough shared understanding of the exchange.

http://bettymartin.org/download-wheel/

[-]Gunnar_Zarncke4y20

From my Anki deck:

Mental play or offline habit training is...

...practicing skills and habits only in your imagination.

Rehearsing motions or recombining them.

Imagine some triggers and plan your reaction to them.

This will apparently improve your real skill.

Links:

https://en.wikipedia.org/wiki/Motor_imagery

http://www.bulletproofmusician.com/does-mental-practice.../

http://expertenough.com/1898/visualization-works

[-]Gunnar_Zarncke4y20

From my Anki deck:

Aaronson Oracle is a program that predicts the next key you will type when asked to type randomly and shows how often it is right.

https://roadtolarissa.com/oracle

Here is Scott Aaronson's comment about it:

In a class I taught at Berkeley, I did an experiment where I wrote a simple little program that would let people type either “f” or “d” and would predict which key they were going to push next. It’s actually very easy to write a program that will make the right prediction about 70% of the time. Most people don’t really know how to type ra

... (read more)

[-]Gunnar_Zarncke4y20

Slices of joy is a habit to...

feel good easily and often.

Trigger Action Plan:

Some small slice of good happens
Notice it consciously.
Enjoy it in a small way.

This is a trigger, a routine, and a reward — the three parts necessary to build a habit. The trigger is the pleasant moment, the routine is the noticing t, and the reward is the feeling of joy itself.

Try to come up with examples; here are some:

- Drinking water.

- Eating something tasty

- Seeing small children

- Feeling of cold air

- Warmth of sunlight

- Warmth of water, be it bathing, dishwashing, etc.

... (read more)

[-]Gunnar_Zarncke4y20

Refreshing your memory:

What is signaling, and what properties does it have?

- signaling clearly shows resources or power (that is its primary purpose)

- is hard to fake, e.g., because it incurs a loss (expensive Swiss watch) or risk (peacocks tail)

- plausible deniability that it is intended as signaling

- mostly zero-sum on the individual level (if I show that I have more, it implies that others have less in relation)

- signaling burns societal resources

- signaling itself can't be made more efficient, but the resources spent can be used more efficiently in soc

... (read more)

[-]Gunnar_Zarncke4y20

What is the Bem Test or Open Sex Role Inventory?

It is a scientific test that measures gender stereotypes.

The test asks questions about traits that are classified as feminine, masculine, and neutral. Unsurprisingly, women score higher on feminine, and men on masculine traits but Bem thought that strong feminine *and* masculine traits would be most advantageous for both genders.

My result is consistently average feminity, slightly below average masculinity. Yes really. I have done the test 6 times since 2016 and the two online tests mostly agree. And it fits:

... (read more)

[-]Gunnar_Zarncke4y20

What is a Blame Hole (a term by Robin Hanson)?

Blame holes in blame templates (the social fabric of acceptable behavior) are like plot holes in movies.

Deviations between what blame templates actually target, and what they should target to make a better (local) world, can be seen as “blame holes”. Just as a plot may seem to make sense on a quick first pass, with thought and attention required to notice its holes, blame holes are typically not noticed by most who only work hard enough to try to see if a particular behavior fits a blame template. While many ar

... (read more)

[-]Gunnar_Zarncke4y20

Leadership Ability Determines a Person's Level of...

Effectiveness.

(Something I realized around twelve years ago: I was limited in what I could achieve as a software engineer alone. That was when I became a software architect am worked with bigger and bigger teams.)

From "The 21 Irrefutable Laws of Leadership

By John C. Maxwell":

Factors That Make a Leader

1) Character – Who They Are – true leadership always begins with the inner person. People can sense the depth of a person's character.

2) Relationships – Who They Know – with deep relationships with the right ... (read more)

[-]Gunnar_Zarncke4y20

To achieve objective analysis, analysts do not avoid what?

Analysts do not achieve objective analysis by avoiding preconceptions; that would be ignorance or self-delusion. Objectivity is achieved by making basic assumptions and reasoning as explicit as possible so that they can be challenged by others and analysts can, themselves, examine their validity.

PS. Any idea how to avoid the negation in the question?

[-]Gunnar_Zarncke4y20

I started posting life insights from my Anki deck on Facebook a while ago. Yesterday, I stumbled over the Site Guide and decided that these could very well go into my ShortForm too. Here is the first:

Which people who say that they want to change actually will do?

People who blame a part of themselves for a failure do not change.
If someone says, "I've got a terrible temper," he will still hit. If he says, "I hit my girlfriend," he might stop.
If someone says, "I have shitty executive function," he will still be late. If he says, "I broke my

... (read more)

[-]Gunnar_Zarncke4y20

My son (15) shared this Instagram version of Newcomb's Problem.

[-]Gunnar_Zarncke4y20

I'm looking for a post on censorship bias (see Wikipedia) that was posted on here on LW or possibly on SSC/ACX but a search for "censorship bias" doesn't turn up anything. Googling for it turns up this:

https://www.theatlantic.com/business/archive/2012/05/when-correlation-is-not-causation-but-something-much-more-screwy/256918/

Anybody can help?

[-]Gunnar_Zarncke4y20

Philosophy with Children - In Other People's Shoes

"Assume you promised your aunt to play with your nieces while she goes shopping and your friend calls and invites you to something you'd really like to do. What do you do?"

This was the first question I asked my two oldest sons this evening as part of the bedtime ritual. I had read about Constructive Development Theory and wondered if and how well they could place themselves in other persons' shoes and what played a role in their decision. How they'd deal with it. A good occasion to have some philosophical t... (read more)

[-]Gunnar_Zarncke4y*20

Philosophy with Children - Mental Images

One time my oldest son asked me to test his imagination. Apparently, he had played around with it and wanted some outside input to learn more about what he could do. We had talked about https://en.wikipedia.org/wiki/Mental_image before and I knew that he could picture moving scenes composed of known images. So I suggested

a five with green white stripes - diagonally. That took some time - apparently, the green was difficult for some reason, he had to converge there from black via dark-green
three mice
three mice,

... (read more)

[-]Gunnar_Zarncke5y20

Origins of Roles

The origin of the word role is in the early 17th century: from French rôle, from obsolete French roule ‘roll’, referring originally to the roll of paper on which the actor's part was written (the same is the case in other languages e.g. German).

The concept of a role you can take on and off might not have existed in general use long before that. I am uncertain about this thesis but from the evidence I have seen so far, I think this role concept could be the result of the adaptations to the increasing division of labor. Before that peop... (read more)

[-]Gunnar_Zarncke5y20

The Cognitive Range of Roles

A role works from a range of abstraction between professions and automation. In a profession one person masters all the mental and physical aspects of trade and can apply them holistically from small details of handling material imperfections to the organization of the guild. At the border to automation, a worker is reduced to an executor of not yet automated tasks. The expectations on a master craftsman are much more complex than on an assembly-line worker.

With more things getting automated this frees the capacity to automate m... (read more)

[-]Gunnar_Zarncke5y20

When trying to get an overview of what is considered a role I made this table:

Type of role	Example	Purpose	Distinction	CDF Level
(Children's) play acting	Cop, Father	Play, imitation, learning	Shallow copy, present in higher animals	Impulsive (1) to Instrumental (2)
Social role	Mother, Husband	Elementary social function	Since the ancestral environment, closely moderated by biological function	Instrumental (2) to Socialized (3)
Occupation (rarely called role but shares traits)	Carpenter	Getting things done in a simple society	Rarely changed, advancement possible (appren

... (read more)

[-]Gunnar_Zarncke5y20

In any sizable organization, you can find a lot of roles. And a lot of people filling these roles - often multiple ones on the same day. Why do we use so many and fine-grained roles? Why don’t we continue with the coarse-grained and more stable occupations? Because the world got more complicated and everybody got more specialized and roles help with that. Division of labor means breaking down work previously done by one person into smaller parts that are done repeatedly in the same way - and can be assigned to actors: “You are now the widget-maker.” This w... (read more)

[-]Gunnar_Zarncke5y20

What are the common aspects of these labor-sharing roles (in the following called simply roles)?

One common property of a role is that there is common knowledge by the involved persons about the role. Primarily, this shared understanding is about the tasks that can be expected to be performed by the agent acting in the role as well as about the goals to be achieved, and limits to be observed as well other expectations. These expectations are usually already common knowledge long beforehand or they are established when the agent takes on the role.

The s... (read more)

[-]Gunnar_Zarncke7mo*1-4

[provided as is, I have no strong opinion on it, might provide additional context for some]

The Social Radars episode on Sam Altman:

Carolynn and I have known Sam Altman for his whole career. In fact it was Sam who introduced us. So today's Social Radars is a special one. Join us as we talk to Sam about the inside story of his journey from Stanford sophomore to AI mogul.

Link to podcast, Tweet

I'm posting this specifically because there is some impression that OpenAI and Sam Altman are not good but self-interested, esp. voiced by Zvi.

One of the thi

... (read more)

[-]Gunnar_Zarncke2y10

I have enabled Reacts for my Shortform. Let's see how it goes.

Moderation Log