I'm not sure I understand how cults are examples of taking an idea seriously, surely a cult is a complex of ideas - not any single one, some of which one can take seriously and others not so (in relgions there's debates about Hyperdispensationalism and patripassianism which show that even within the complex of ideas, different ones can be taken seriously. Not to mention a la carte Catholics and reformists ) - and that the chief mechanism by which people become subsumed into cults has nothing to do with reason or logical arguments but social support (or coercion) irrespective of the recruits belief?

The feedback loop is very different then and operates not on ideas but a whole host of different mechanisms. (Feelings of belonging, feelings of personal importance, no longer a need to 'search' or 'question' existential matters). These don't require ideas to be taken seriously at all.

Again, on the macro scale I can take seriously the idea of... I dunno... Lamarckism. But even if I seriously investigate it, give it the benefit of the doubt, I'm not really in a position to test it in the sense that it's a macro idea and not something whcih will affect my everyday routine (like Polyphasic sleeping). Even if I later on have children and try to change my behavior to elicit certain traits in those children, the lag time between when I can confirm it is many years. 

it feels like arguments just being obviously very compelling, so you'll notice nothing wrong if it happens to you.

Does this only apply on the macroscale, say, ideas concerning ASI or Economic frameworks? Because it feels like if I take a very personal level idea seriously, let's take polyphasic sleeping. If I take it seriously and implement it - sure I won't get thrown into the East River but I should notice if something wrong happens to me, and rather quickly.

Solution really seems to be: tight feedback loops?
 

Great post, at least for a non-statistics-literate person like me. The Scots-Irish example (in spite of Arran McCutcheon's nitpick, which I'm all here for) makes concrete why this phenomenon is happening.  The lizardman's constant shows up again?

There's another approach that has experimented with something similar:
https://hanlab.mit.edu/blog/streamingllm
 

We found models dump massive attention onto the first few tokens as "attention sinks"—places to park unused attention since softmax requires weights to sum to 1. Our solution, StreamingLLM, simply keeps these first 4 tokens permanently while sliding the window for everything else, enabling stable processing of 4 million+ tokens instead of just thousands.

Interesting, it would be fun to try it with the Claude Tokenizer

I don't think the Cyrillic text would map to any common tokens, since the output is essentially the result of a substitution cipher, the key being the keyboard mappings.

I don't understand. surely it has been exposed to training resources that contain, say, Serbian which is written in both Latin and Cyrillic. And more relevant: news articles that have transliterations of Anglophone celebrity names and places:

Дэвід Бекхэм (David Beckham)
Стенлі Кубрик (Stanley Kubrick)
Лінкольншир (Lincolnshire)

Why wouldn't these map to common tokens?
 

I remember reading that LLMs are especially good at Caesar Ciphering which might explain how they can transliterate Cyrillic into Latin, this is probably an unintended side effect of the way embeddings work since what is not encoded isn't the English sentence, but the relative positions of the vectors each token is converted to.

To put it another way, your Cyrillic gibberish and your Latin alphabet are, in embedding space, very very similar. It would be interesting to play around with reverse writing and one-letter-up.

Like asking:

xibu xbt uif jodbo fdpopnz mjlf?

Although my suspision is since that, phonetically speaking, the cyrillic version of your sentence would map to more common tokens than my one-letter-up rendition, perhaps you will experience wildly different results?
 

Time is not, in terms of experience, uniform. Therefore even with extra time, priorities can vary. People tend to have 5 hours of peak productivity a day - this doesn't mean they couldn't be generally more productive with additional off-peak hours, but it does mean that priories vary depending on what 'kind' of hour we're hypothesizing.

For example, folding clothes and putting them away I can do off-peak. However, I don't like driving my car too late at night - even though I prefer the lack of traffic - because I don't like driving with diminished alertness. As such, my driving habits and how I structure my day may not change that much.

You may find that with more off-peak hours my top priorities during peak hours wouldn't change, but my off-peak and lower priorities would.

I always assumed it's because in western society,  the career of "artist" is smeared and sneered at for being "not a real job". My creative friends often mention the risk of being "taken advantage of" when it comes to payment and remuneration - that since they love what they do, they should be expected to "do it for free" or be underpaid.

And at society at large there's this idea that you get paid to suffer - and jobs aren't expected to be appealing or fun. So you shouldn't expect to be paid for making art. This does dovetail as a cause of financial precarity you're alluding to - but I believe there's more malice or disdain behind the reaction than just neutral risk assessment. Therefore when A.I. comes and gobbles up paid opportunities for artists, the view is "well that wasn't a serious job anyway - it's just a hobby, a passion. You don't get paid to do what you love, you get paid to do what you hate." rather than "it's a valid job, but good luck making a living"

To be honest, I've never actually looked into this to back it up.  I'm making a lot of assumptions here and putting a lot of thoughts/words into a nebulous group known as "society". The closest is I've read some historical analysis about the shift in views of genius (including artistic) from the romantic era and into the industrial age that underscore a shifting sentiment from being driven by passion or inspiration (passive), into patience and discipline (active).

What are the common counterpoints to “there’s nothing I, personally, can do about p(doom) or various X-Risks so I just don’t think about it”?

For example, I have a liberal arts education, no social influence, no platform I can beat the drum about specific risks from. I feel no more likely to affect change in, say, development or policy about misaligned super intelligent A.I. than I do about something like the cost of rent in my country, or the legislation around motorized scooters.