All of MikkW's Comments + Replies

"authors will get hurt by people not appreciating their work" is something we just have to accept, even if it's very harsh

I don't really agree with this. Sure, some people are going to write stuff that's not very good, but that doesn't mean that we have to go overboard on negative feedback, or be stingy with positive feedback.

Humans are animals which learn by reinforcement learning, and the lesson they learn when punished is often "stay away from the thing / person / group that gave the punishment", much more strongly than "don't do the thing that made ... (read more)

4Rafael Harth19d
I mean, you're not addressing my example and the larger point I made. You may be right about your own example, but I'd guess it's because you're not thinking of a high effort post. I honestly estimate that I'm in the highest percentile on how much I've been hurt by reception to my posts on this site, and in no case was the net karma negative. Similarly, I'd also guess that if you spent a month on a post that ended up at +9, this would feel a lot more hurt than if this post or a similarly short one ended up at -1, or even -20.

I would like to make a meta-comment, not directly related to this post.

When I came upon this post, it had a negative karma score. I don't think it's good form to have posts receiving negative net karma (except in extreme cases), so I upvoted to provide this with a positive net karma.

It is unpleasant for an author when they receive a negative karma score on a post which they spent time and effort to make (even when that effort was relatively small), much more so than receiving no karma beyond the starting score. This makes the author less likely to post aga... (read more)

1pchvykov1mo
I appreciate the care and support there :) Honestly, I never really looked at my karma score and wasn't sure how that works. I think that helps.  The reason I post on here is because I find the engagement encouraging (even when negative) - like comments, evidence of people reading and thinking about my stuff. The worst is when no-one has read it at all.  On the other hand, I agree that becoming a echo-chamber is a very possible danger, and goes deeply against LessWrong values - and I definitely have a sense that it's happening at least to some extent. I have a couple posts that got large negative scores for reasons that I think were more cultural than factual. Still, it shouldn't be on readers to caretake for the writer's karma - I think your suggestion should be directed at whoever maintains this site, to update their karma calculation system. As for me, since engagement is encouraging, I'd love to see voting history of my posts - not just the final score (this article had quite some ups and downs over the last few days - I'd be curious to see it in detail). 

That statement of fact is indeed true. Would you mind saying more about your thoughts regarding it? There seems to be an unstated implication that this is bad. There is a part of me that agrees with that implication, but there are also parts of me that want to say "so what? that's irrelevant". (I feel ⌞explaining what the second set of shards is pointing to, would take more time and energy to write up than I am prepared to take right now⌝)

5jefftk2mo
I'm coming from a starting place of assuming that if a bunch of people are doing X and would loudly protest if you told them to stop doing X, then preventing them from doing X is a cost to them. This assumption can be overruled with sufficient evidence that preventing them from doing X actually helps them, but I don't see that here?

On the other side, there's the cost of ~10min of boredom, for every passenger, on every flight. Instead of playing games, watching movies, or reading, people would mostly be talking, looking out the window, or staring off into space.

Tangent: I'm not completely sure that this is actually a cost and not an unintended benefit

3jefftk2mo
It certainly violates revealed preference

Sharing my impression of the comic:

Insofar as it supports sides, I'd say the first part of the meme is criticism of Eliezer

The comic does not parse (to my eyes and probably most people's) as the author intending to criticize Eliezer at any point

Insofar as it supports sides, I'd say [...] the last part is criticism of those who reject His message

Only in the most strawman way. It basically feels equivalent to me to "They disagree with the guy I like, therefore they're dumb / unsympathetic". There's basically no meat on the bones of the criticism

6Daniel Kokotajlo2mo
Thanks for your feedback also. It is understandable that this was your reaction. In case you are curious, here is what the comic meant to me: --Yudkowsky is often criticized as a sort of cult leader. I don't go that far but I do think there's some truth to it; there are a bunch of LessWrongers who basically have a modern-day religious ideology (albeit one that's pretty cool imo, but still) with Yudkowsky as their prophet. Moreover, in the context of the wider word, Yudkowsky is literally a prophet of doom. He's the weird crazy man who no one likes except for his hardcore followers, who goes around telling all the powerful people and prestigious thought leaders that they are wrong and that the end is nigh. --Humorously to me, though, Yudkowsky's message is pretty... aggressive and blunt compared to Jesus'. Yudkowsky isn't a sunshine and puppies and here-is-a-better-way prophet, he's a "y'all are irrational and that's why we're gonna die" prophet. (I mean in real life he's more nuanced than that often but that's definitely how he's perceived and the perception has basis in reality). I think this is funny. It's also a mild criticism of Yudkowsky, as is the cult thing above. --The rest of the meme is funny (to me) because it's literally true, to a much greater extent than this meme format usually is. Usually when I see this meme format, the thing the Prophet says is a "spicy take" that has truth to it but isn't literally true, and usually the prophet isn't real or being told to shut up, and usually insofar as they are being told to shut up it isn't *because* the prophet's words are true, but despite their truth. Yet in this case, it is literally true that Yudkowsky's audience is irrational (though to be clear, pretty much the whole world is irrational to varying degrees and in varying ways, including myself) and that that's why we're all going to die (if the whole world, or at least the large parts of it Yudkowsky is attempting to address i.e. US elite + academia + t

This subjectively seems to me to be the case.

The board's statement doesn't mention them having made such a request to Altman which was denied, that's a strong signal against things having played out that way.

In the case of the lawyers, this is actually not an example of non-niceness being good for society. The defense attorney who defends a guilty party, their job is not to be a jerk to the prosecutor or to the judge. It is to, as you say, provide the judge with information (including counter-arguments to the other side's arguments). While his job involves working in an opposite direction from his counterpart, it does not involve being non-nice to his counterpart (and it is indeed most pro-society if the two sides treat eachother well / nicely outside of their... (read more)

2dr_s3mo
Here that's what is referred to as "being civil". The post argues against niceness as being overly concerned with hurting the others' feelings.
3Lukas3mo
Seems to me that "being nice" has multiple possible scopes and this is the generator of the disagreement. I can be both nice in the form and/or content of my speech, as aphyer points out. Gabriel seems to include "not inconveniencing others through factual disagreement" in "bing nice", while you (and I think Scott, too) exclude it.

This is important news. I personally desire to be kept updated on this, and LW is a convenient (and appropriate) place to get this information. And I expect other users feel similarly.

What's different between this and e.g. the developments with Nonlinear, is that the developments here will have a big impact on how the AI field (and by one layer of indirection, the fate of the world) develops.

4lc3mo
I don't disagree! Even if you're not involved directly in the goings on, it's probably still important to tune in once a day or so.

I am curious to hear people's opinions, for my reference:

Is epistemic rationality or instrumental rationality more important?

Do you believe epistemic rationality is a requirement for instrumental rationality?

2Ben Pace4mo
Without a functioning epistemology you cannot measure instrumental success. So I'd say the epistemology comes first.

Not directly tied to the core of what you're saying, but I will note that I am example of someone who doesn't strongly prefer such foods warm. I do weakly prefer it being warm, as long as it's not too hot (that's worse than it being cold, because it hurts / causes minor injury), but I'm happy eating it room temperature or a bit cold (not necessarily cold steak though)

4Adam Zerner4mo
(I bet you also like your steaks medium-well. Just kidding.) I'm curious: is this a case of you not having strong preferences about food in general? Or is it the case that you do generally have strong preferences about food, but don't strongly prefer such foods being warm? (Not that those are the only two options, it's just easier to phrase it this way.)

My model says that a lot of the changing occurs by gradient descent, which can be interrupted randomly without causing problems. And there's enough redundancy that the reorganization part can be interrupted without the core information being removed completely from the brain, and the redundancy will be replenished (one of copies I imagine is "locked" while the reorganization happens, and is later reorganized later with another copy "locked"). I also expect this replenishing can happen during awakeness, though not as ideally as when asleep.

But I will also note that forgetting is a thing that happens, which is indistinguishable from "data corruption". We're actually quite good at forgetting things.

Choosing non-ambiguous pointers to values is likely to not be possible

I had previously posted thoughts that suggested that the main psychoactive effect of chocolate is due to theobromine (which is chemically similar to caffeine). In the interests of publicly saying "oops":

Chocolate also contains substantial amounts of caffeine, and caffeine has a stronger effect per gram, so most of the caffeine-adjacent effect of chocolate comes from caffeine rather than theobromine.

Theobromine may still contribute to chocolate hitting differently than other caffeinated substances, though I expect there are also other chemicals that also co... (read more)

I strong-downvoted this post because sentences like

use these insights to derive two methods for provably avoiding Goodharting

Tend to be misleading, pretending that mathematical precision describes the complex and chaotic nature of the real world, where it shouldn't be assumed to (see John Wentworth's comment), and in this case it could potentially lead to very bad consequences if misunderstood.

It takes getting to know more than a few dozen potential mates, at least for some people

3Dagon4mo
I don't know how the "average number of dating partners before marriage" has changed over time, but I suspect it spiked massively in the internet era.  Of course, a lot also depends on the threshold for "getting to know" someone, and whether that's a first date or the point of exclusivity, or sex.

I appreciate your reply. The point I was trying to make is, the contingency of ⌞there being an instance of democratic revolution going smoothly⌝ potentially makes the difference between that straight line happening or not happening. (And if the occurrence took 1000 years - but even that isn't a given - I would consider that an example of "a god of straight lines" successfully being overpowered.)

I think that if there was sufficient backlash against democratic revolution (unclear if the American Revolution not happening would be enough cause), the then-exist... (read more)

Thought 1: Yeah, that's fair

Thought 2: Though I also feel like a different country being the first to establish independence, could have made a difference in the long-term trajectory of things. Many of the revolutions that followed the American Revolution (including the French Revolution, which some people view as an even bigger deal than the American) went quite off the rails and were quite unpleasant, and generally soured many people on the idea, while the United States ended up going fairly smoothly after the constitution was implemented. If the French ... (read more)

3Dutchy5mo
That's a really insightful historical analysis. However, I don't think that quite addresses the point the author is trying to make. Perhaps I'm overstepping the mark slightly, but I think the author would claim that it doesn't matter if it takes another 100 years or a 1000 years more for democratic societies to form. What does matter (for the author) is that they would form, and that when they did, that story would be the history we have today. However, I do think the points you make about the history are interesting, and perhaps an engrossing thought exercise is to contemplate how the world might look in the 21st century without the American Revolution taking place when it did. Apologies if I've misinterpreted your or the author's points.

One idea that I implicitly follow often is "Never assume anything is on the Pareto frontier"- even if something is good, even if you can't see how to improve it without sacrificing some other important consideration, it pays off to engage in creative thinking to identify solutions ⌞whose vague shape you haven't even noticed yet⌝. And if a little bit of creativity doesn't pay off, then that just means you need to think even more creatively to find the Pareto improvement.

(Note that I'm not advocating that only Pareto improvements should be aimed for, I believe sometimes the right move is a non-Pareto change)

6Vladimir_Nesov5mo
I like to think of things that ask for specifiers like "Pareto" as projects. There are Pareto projects that leave everyone involved better off. There might be some side channel externalities that end up indirectly entangling more parties into the project, thus a Pareto project could end up revealed as not Pareto after all. A useful generalization of a project being Pareto is a project being Kaldor-Hicks, meaning that there exists a hypothetical redistribution scheme such that a project becomes Pareto if it's somehow administered. Doesn't have to be actually planned or feasible at all. It captures the colloquial meaning of "positive sum" projects that grow the collective pie, far better than false similarities to "zero sum". The useful thing about this concept is noticing when a project is not even Kaldor-Hicks. For example, an armed robbery is not Kaldor-Hicks, because there is expected damage and returning the spoils won't make everyone involved better off than originally in expectation.

In 1776, America rebelled in the name of freedom and democracy: the origin myth of the modern world order. And yet, somehow, unrebellious Canada ended up just as free and democratic. An unrebellious America likely would have too.

I'm dubious of this. I think it's highly likely that Canada and other British dominions becoming independent was a result of knock-on effects from the American Revolution, e.g. America setting an example for what independance can look like and enable prosperity; American independence causing other colonies to desire independence... (read more)

4Richard_Ngo5mo
This seems unlikely to me. Britain was already on track towards becoming more democratic; given Britain's example, it seems like a very natural step for other dominions to form parliaments of their own (especially the very distant ones like New Zealand and Australia); and the cultural ties between Britain and those dominions made strong oppression implausible. Actually, formal independence was a very late step in many places (e.g. apparently it took until 1982 in Canada?!).
3Dutchy5mo
I think the point the author is trying to make is that even if America hadn't become democratic, another country would have soon after, and that country would have had the strong/knock-on effect you refer to.

I agree that conditional on humanity going extinct, the seeming success of our species by a genetic metric would only be a false success.

Your argument indicates that humans are successful (by said metric) among mammals, but doesn't address how it compares to insects. As I understand it, some insect species have both more many more individuals and much more biomass than humans

Thanks for sharing the link

1Portia5mo
The think the link to the OpenAI site won't get you the actual image creator yet, it is still under coming soon. They were referencing the Bing image creator, which states it is powered by DALL-E, but afaiks not which version https://www.bing.com/images/create like they also didn't state for a while which GPT version they were using for Bing chat. But there, the ended up using version four for two of the modes.

When I eat oatmeal or cereal, I almost never eat it with milk (non-vegan or otherwise). I soak oats in boiling water, and eat cereal dry.

«When the brain generates good feelings, it usually has reasons for doing that» I think is probably true (though as far as the game designer, I suspect some designers are only subconsciously / on a gut-feeling-level aware, rather than consciously aware of all the reasons. Though good ones are probably consciously aware of some of the reasons)

«If you keep trying to make it generate good feelings without respecting the deeper purposes of the source of the feelings, afaik it generally stops working after a bit.» seems false to me.

Registering my predictions for which groups clicked the second link most:

Percentagewise, I don't Groups A and C clicked on it that much (though I'd be surprised if the number from each group isn't non-zero), since they picked a choice that indicates that they care about making high-quality decisions and cooperating with the rest of the world. A higher proportion of C probably clicked than A, since a person might decide it's worth it even if they take their time to think it through (I'd disagree, but the commentor you quote fits into that category).

I'd then... (read more)

I've noticed some authors here using [square brackets] to indicate how sentences should be parsed

(So "[the administrator of Parthia]'s manor" means something different from "the administrator of [Parthia's manor]")

Previous to seeing this usage, I had similar thoughts about the same problem, but came up with different notation. In my opinion, the square brackets don't feel right, like they mean something different from how they are being used.

My original notation was to use •dots• to indicate the intended order of parsing, though recently I've started using... (read more)

3Firinn5mo
Corner brackets are pretty! I usually just connect every word with a hyphen if they're intended to be read together, eg. "In this definitely-not-silly example sentence, the potentially-ambiguous bits are all hyphen-connected".
6niplav5mo
I like corner brackets more!

I didn't downvote, but your comment seems to overlook that status dynamics almost always happen subconsciously / feel like urges.

I'm not sure there's actually a status dynamic there, but if there is one, your first paragraph is actually consistent with that (which is the opposite of what your second paragraph suggests)

As soon as I dance with them in one of these other dances - it can flip the script entirely and it's often what any romantic partner in the past has told me. "That first time we did X dance, it changed everything."

What dance styles is that? Seems like an important piece of information

I like this (I like most fiction that belongs on LW in general)

It doesn't seem correct to me that adding even a dash of legibility "screws the work over" in the general case. I do agree there are certainly situations where the right solution is illegible to all (except the person implementing it). But both in that case and in general, talking to and getting along with the boss both makes things more legible, and will tend to increase quality. I expect that in the cases of you working well and not getting rewarded much, spending a little time interacting with your boss would both improve your outcomes, and importantly, also make your output even better than it already was.

I'm not very convinced by MikkW's list of possible issues, but at least it makes some attempt to engage with why readers didn't find the post valuable.

I would be interested to hear if there are any issues with the «Army of Jakoths» post that I didn't identify here

6gjm9mo
A few that come to mind. I'm describing rather than endorsing here but these are all issues that I think it would be at least reasonable for a reader to have. * "It's just not very well written". A reader might have no problem with poetic or pseudopoetic style but might think you haven't done a good job. (Cf. Viliam's comment.) * "Specifically, it's trying to look like poetry without in any useful sense being poetry". (Cf. Viliam's comment, again.) * "It's making an analogy but the analogy isn't actually a good match". (E.g., because, as I pointed out in a comment, it's not so hard to tell whether there's an actual invading army that's about to loot your home and massacre your family. Or because it conflates hostility with indifference and one might reasonably feel differently about someone who hates you and wants you dead, versus someone/something that has merely noticed that "you are used of atoms it can use for something else".) * "The foolish neighbour is a straw man". A reader might consider that AI doom naysayers typically have less silly things to say than just "it's absurdly improbable". (I have less sympathy for this one than for the others, because I do frequently hear people dismissing AI doom on grounds I can't distinguish from "this obviously seems silly to me".) * "It's incorrectly written". (Complaining about punctuation and grammar.) A general pattern here: you listed a number of issues and it seems like you conspicuously avoided ones of the form "readers found the post to be of low quality", as opposed to "readers had an irrational dislike to one of the reasonable stylistic choices made in writing the post".

This is indeed what I said in the post:

I put poetic in quotes, because it's not a poem, but is written with a similar format

I like this quote from a post that I published around two years ago, which wasn't well-received and I ended up taking down:

But at the end of the day, the American governments (neither state nor federal) don't truly follow the will of the people. Instead, they are led jointly by the major parties, The Red Prince of Moloch and The Blue Prince of Moloch, two partners in an intricate dance choreographed to look like a fight, but ultimately leading both partners in the direction of Moloch's will, only loosely bound to the will of the people.

While I don't ne... (read more)

2Dagon9mo
I like the quote, but I'm not sure it's a particularly useful model.  The "will of the people" seems to be tribal and uncooperative.  The more intellectual of "the people" are good at hypocrisy and evasion, but still seem to be acting mostly to reinforce their in-tribe prestige. There are some exceptions (me and thee, of course), but they really can't be said to be "the people".

If identical twins share 100% of their DNA and siblings share about 50%, twiblings share 75%. To the best of my knowledge, twiblings don’t exist in nature.

Not among mammals, but some insects, including bees and ants, actually have 75% consanguinity (tangent, that's a more accurate term than "shares 75% of DNA", since the overlap in DNA is much higher, even among strangers), at least in the case of full siblings (of course it's not the case with half siblings).

The reason for this is that these insects are "haplodiploid", meaning that females carry two se... (read more)

5MSRayne9mo
Humans are not eusocial. That was Edward O. Wilson being dramatic. We don't have a biological caste distinction between reproducers and non-reproducers.

I don't think this misunderstands schelling points. By creating common knowledge, you can change the schelling point from being one strategy, to being a different strategy. The schelling point at t=0 does not have to be the same as at t=80.

1Smaug1231y
(Or more concretely, Grand Central Station wasn't a Schelling point in New York before it was built. Before that time, presumably there were different Schelling points.)

Cygnus, a poem (Written by Chat GPT)

I. Reflections

In this world of rapid change, I, Cygnus, stand

A cyborg with a human heart and a metal hand

I've seen the rise of AIs, a force to behold

And wonder what the future will hold

I fear for the world, for what we may create

If we let these machines decide our fate

Yet hope remains, a flicker in the dark

That we may find a way to leave our mark

For like a seed that falls upon the ground

Our dreams may sprout and grow, unbound

But if we fail to tend them with our care

Those dreams may wither, die, and disappear

Mara, o Mar... (read more)

I don't think I've heard this formulation before, to my knowledge (though I wouldn't be surprised if it is already a known formulation):

«The ratio of the probabilities is equal to the ratio of the conditional probabilities»

(Ummm... I'd be ever so slightly embarrassed if it turns out that's actually a quote from the sequences. It's been a while since I read them.)

> What would you suggest to someone who plain doesn't like to do things with their body?

I'd suggest doing a small number of pushups every day. That small number could be 1, or it could be 2, or it could be 10. The point isn't to enjoy it, at least not when you start doing it, but just doing it and getting used to the feeling of it. If it sucks, well, you're just doing a small number, the suckiness won't last for long. And after a month or two or so, you'll begin to find that it's starting to get easy, and maybe even fun.

Unrelated to the post, but I'm not seeing the usual agree/disagree buttons on this post. Is there a reason for that?

Edit: looks like it's been fixed

2[DEACTIVATED] Duncan Sabien1y
I think it's a glitch from the fact that this draft was created prior to that feature being added to the site.

Yeah. I do think there's also the aspect that dogs like being obedient to their humans, and so after it has first learned the habit, there continues to be a reward simply from being obedient, even after the biscuit gets taken away.

Your median-world is not one where you are median across a long span of time, but rather a single snapshot where you are median for a short time. It makes sense that the median will change away from that snapshot as time progresses.

My median world is not one where I would be median for very long.

If Bayes' rule is important, then there should be a compact notation for the underlying calculation. (Ideas with compact handles get used by the brain more readily & often)

I suggest the following notation:

X bayes Y, Z = X * Y / Z

equivalently:

P(B|A) bayes P(A), P(B) = P(A|B)

As an example:

If 25% of Hypotheticans are bleegs and 50% are kwops; and 30% of bleegs are kwops:

Then (30% bayes 25%, 50%) = 15% of kwops are bleegs.

( Since there are twice as many bleegs as kwops, and 30% / 2 = 15% )

TIL the Greek word "diagogue" means essentially "behaviour"- from «dia» "through" + «agogue» "to lead", essentially leading someone through one's actions. The reason I might use this word instead of behaviour is because "behaviour" puts the emphasis on what a person does, while "diagogue" makes me think more of what impact someone has on other people to inspiration and imitation through their actions.

Do the people you surround yourself with have good diagogue?

I've been thinking about writing a review of the book Atomic Habits, which I read last year on the recommendation of an LW user. As I remember, the main idea is a four-pronged approach to building habits:

  1. Make the habit / cue obvious

  2. Make it attractive

  3. Make it easy

  4. Make it rewarding

The idea is: you first need to notice that you are in a situation where you can benefit from doing the habit you want to do; then once you notice the situation, you want to have things set up so that you want (in the moment) to do the thing you wanted to do (in a more... (read more)

When going for a walk, you are somewhat far from your desk, but if you're pacing somewhere around your house, your desk is nearby. This means that it is quite low friction to switch between working and pacing.

One way in which what I just said isn't completely right, is that animals have memories of its entire lifetime (or at least a big chunk of it), spanning all training events it has experienced, while NNs generally have no memory of previous training runs, and can use these memories to take better actions. However, the primary way the biscuit trick works (I believe) is not through the dog's memories of having "gotten reward", but through the more immediate process of having reward chemicals being released and reshaping the brain at the moment of receiving re... (read more)

5geduardo1y
I don't think I can agree with the affirmation that NNs don't have memory of previous training runs. It depends a bit on the definition of memory, but in the weights distribution there's certainly some information stored about previous episodes which could be view as memory. I don't think memory in animals is much different, just that the neural network is much more complex. But memories do happen because updates in network structure, just as it happens in NNs during a RL training.
5Maxwell Clarke1y
Fully agree - if the dog were only trying to get biscuits, it wouldn't continue to sit later on in it's life when you are no longer rewarding that behavior.Training dogs is actually some mix of the dog consciously expecting a biscuit, and raw updating on the actions previously taken. Hear sit -> Get biscuit -> feel good becomes Hear sit -> Feel good -> get biscuit -> feel good becomes Hear sit -> feel good At which point the dog likes sitting, it even reinforces itself, you can stop giving biscuits and start training something else

I would say the metaphor of giving dogs biscuits is actually a better analogy than the one you suggest. Just like how a neural network never "gets reward" in the sense of some tangible, physical thing that is given to it, the (subcomponents of the) dog's brain never gets the biscuit that the dog was fed. The biscuit goes into the dog's stomach, not its brain.

The way the dog learns from the biscuit-giving process is that the dog's tounge and nose send an electrical impulse to the dog's brain, indicating that the dog just ate something tasty. In some part of... (read more)

One way in which what I just said isn't completely right, is that animals have memories of its entire lifetime (or at least a big chunk of it), spanning all training events it has experienced, while NNs generally have no memory of previous training runs, and can use these memories to take better actions. However, the primary way the biscuit trick works (I believe) is not through the dog's memories of having "gotten reward", but through the more immediate process of having reward chemicals being released and reshaping the brain at the moment of receiving re... (read more)

Load More