the gears to ascenscion's Shortform

should I post this paper as a normal post? I'm impressed by it. if I get a single upvote as shortform, I'll post it as a full fledged post.
Interpreting systems as solving POMDPs: a step towards a formal understanding of agency
Martin Biehl, N. Virgo
Published 4 September 2022
Philosophy
ArXiv
. Under what circumstances can a system be said to have beliefs and goals, and how do such agency-related features relate to its physical state? Recent work has proposed a notion of interpretation map , a function that maps the state of a system to a probability dist... (read more)

Everyone doing safety research needs to become enough better at lit search that they can find interesting things that have already been done in the literature without doing so adding a ton of overhead to their thinking. I want to make a frontpage post about this, but I don't think I'll be able to argue it effectively, as I generally score low on communication quality.

comment I decided to post out of context for now since it's rambling:

formal verification is a type of execution that can backtrack in response to model failures. you're not wrong, but formally verifying a neural network is possible; the strongest adversarial resistances are formal verification and diffusion; both can protect a margin to decision boundary of a linear subnet of an NN, the formal one can do it with zero error but needs fairly well trained weights to finish efficiently. the problem is that any network capable of complex behavior is likely to b... (read more)

while the risk from a superagentic ai is in fact very severe, non-agentic ai doesn't need to eliminate us for us to get eliminated, we'll replace ourselves with it if we're not careful - our agency is enough to converge to that, entirely without the help of ai agency. it is our own ability to cooperate we need to be augmenting; how do we do that in a way that doesn't create unstable patterns where outer levels of cooperation are damaged by inner levels of cooperation, while still allowing the formation of strongly agentic safe co-protection?

[tone: humorous due to imprecision]

broke: effective selfishness
woke: effective altruism
bespoke: effective solidarity
masterstroke: effective multiself functional decision theoretic selfishness

a bunch of links on how to visualize the training process of some of today's NNs; this is somewhat old stuff, mostly not focused on exact mechanistic interpretability, but some of these are less well known and may be of interest to passers by. If anyone reads this and thinks it should have been a top level post, I'll put it up onto personal blog's frontpage. Or I might do that anyway if I think I should have tomorrow.

https://metaphor.systems/search?q=cool%20paper%20visualizing%20the%20trajectory%20of%20representations%20in%20the%20process%20of%20training

Modeling Strong and Human-Like Gameplay with KL-Regularized Search - we read this one on the transhumanists in vr discord server to figure out what they were testing and what results they got. key takeaways according to me, note that I could be quite wrong about the paper's implications:

• Multi-agent game dynamics change significantly as you add more coherent search and it becomes harder to do linear learning to approximate the search. (no surprise, really.)
• it still takes a lot of search.
• guiding the search is not hopeless in the presence of noise!
• shallo

index of misc tools I have used recently, I'd love to see others' contributions - if this has significant harmful human capability externalities let me know:

basic:

• linked notes: https://logseq.com/ - alternatives I considered included obsidian, roamresearch, athensresearch, many others; logseq is FOSS, agpl, works with local markdown directories, is clojure, is a solid roam clone with smoother ui, did I mention free
• desktop voice control: https://talonvoice.com/ - patreon-funded freeware. voice control engine for devs. configured with nice code. easier in

btw neural networks are super duper shardy right now. like they've just, there are shards everywhere. as I move in any one direction in hyperspace, those hyperplanes I keep bumping into are like lines, they're walls, little shardy wall bits that slice and dice. if you illuminate them together, sometimes the light from the walls can talk to each other about an unexpected relationship between the edges! and oh man, if you're trying to confuse them, you can come up with some pretty nonsensical relationships. they've got a lot of shattery confusing shardbits a... (read more)

Here's a ton of vaguely interesting sounding papers on my semanticscholar feed today - many of these are not on my mainline but are very interesting hunchbuilding about how to make cooperative systems - sorry about the formatting, I didn't want to spend time format fixing, hence why this is in shortform. I read the abstracts, nothing more.

As usual with my paper list posts: you're gonna want tools to keep track of big lists of papers to make use of this! see also my other posts for various times I've mentioned such tools eg semanticscholar's recommend... (read more)

I've been informed I should write up why I think a particle lenia testbed focused research plan ought to be able to scale to AGI where other approaches cannot. that's now on my todo list.

my reasoning: time is short, and in the future, we discover we win; therefore, in the present, we take actions that make all of us win, in unison, including those who might think they're not part of an "us".

so, what can you contribute?

what are you curious about that will discover we won?

Kolmogorov complicity is not good enough. You don't have to immediately prove all the ways you know how to be a good person to everyone, but you do need to actually know about them in order to do them. Unquestioning acceptance of hierarchical dynamics like status, group membership, ingroups, etc, can be extremely toxic. I continue to be unsure how to explain this usefully to this community, but it seems to me that the very concept of "raising your status" is a toxic bucket error, and needs to be broken into more parts.

the safer an ai team is, the harder it is for anyone to use their work.

so, the ais that have the most impact are the least safe.

what gives?

watching https://www.youtube.com/watch?v=K8LNtTUsiMI - yoshua bengio discusses causal modeling and system 2

hey yall, some more research papers about formal verification. don't upvote, repost the ones you like; this is a super low effort post, I have other things to do, I'm just closing tabs because I don't have time to read these right now. these are older than the ones I shared from semanticscholar, but the first one in particular is rather interesting.

• https://arxiv.org/abs/2012.09313 - Generate and Verify: Semantically Meaningful Formal Analysis of Neural Network Perception Systems (metaphor search for this)
• a metaphor search I used to find some stuff
• https

Yet another ChatGPT sample. Posting to shortform because there are many of these. While searching for posts to share as prior work, I found the parable of predict-o-matic, and found it to be a very good post about self-fulfilling prophecies (tag). I thought it would be interesting to see what ChatGPT had to say when prompted with a reference to the post. It mostly didn't succeed. I highlighted key differences between each result. The prompt:

Describe the parable of predict-o-matic from memory.

samples (I hit retry several times):

1: the standard refusal: I'm ... (read more)

the important thing is to make sure the warning shot frequency is high enough that immune systems get tested. how do we immunize the world's matter against all malicious interactions?

diffusion beats gans because noise is a better adversary? hmm thats weird, something about that seems wrong

Toward a Thermodynamics of Meaning.
Jonathan Scott Enderle.
As language models such as GPT-3 become increasingly successful at generating realistic text, questions about what purely text-based modeling can learn about the world have become more urgent. Is text purely syntactic, as skeptics argue? Or does it in fact contain some semantic information that a sufficiently sophisticated language model could use to learn about the world without any additional inputs? This paper describes a new model that suggests some qualified answers to those questions. By the... (read more)

does yudkowsky not realize that humans can also be significantly improved by mere communication? the point of jcannell's posts on energy efficiency is that cells are a good substrate actually, and the level of communication needed to help humans foom is actually in fact mostly communication. we actually have a lot more RAM than it seems like we do, if we could distill ourselves more efficiently! the interference patterns of real concepts fit better in the same brain the more intelligently explained they are - intelligent speech is speech which augments the user's intelligence, iq helps people come up with it by default, but effective iq goes up with pretraining.

would economic interpretability-to-purchaser align the economy?

https://arxiv.org/abs/2205.15434 - promising directions! i skimmed it!

Learning Risk-Averse Equilibria in Multi-Agent Systems Oliver Slumbers, David Henry Mguni, Stephen McAleer, Jun Wang, Yaodong Yang Download PDF In multi-agent systems, intelligent agents are tasked with making decisions that have optimal outcomes when the actions of the other agents are as expected, whilst also being prepared for unexpected behaviour. In this work, we introduce a new risk-averse solution concept that allows the learner to accommodate unexpected actions by finding the min... (read more)

my question is, when will we solve open source provable diplomacy between human-sized imperfect agents? how do you cut through your own future shapes in a way you can trust doesn't injure your future self enough that you can prove that from the perspective of a query, you're small?

the whole point is to prevent any pivotal acts. that is the fundamental security challenge facing humanity. a pivotal act is a mass overwriting. unwanted overwriting must be prevented, but notably, doing so would automatically mean an end to anything anyone could call unwanted death.

there are opinion clusters in social connection space

neural cellular automata seem like a perfectly acceptable representation for embedded agents to me, and in fact are the obvious hidden state representation for a neural network that will in fact be a computational unit embedded in real life physics, if you were to make one of those.

reminder: you don't need to get anyone's permission to post. downvoted comments are not shameful. Post enough that you get downvoted or you aren't getting useful feedback; Don't map your anticipation of downvotes to whether something is okay to post, map it to whether other people want it promoted. Don't let downvotes override your agency, just let them guide it up and down the page after the fact. if there were a way to more clearly signal this in the UI that would be cool...

oh hell yeah https://www.explainpaper.com/

if status refers to deference graph centrality, I'd argue that that variable needs to be fairly heavily L2 regularized so that the social network doesn't have fragility. if it's not deference, it still seems to me that status refers to a graph attribute of something, probably in fact graph centrality of some variable, possibly simply attention frequency. but it might be that you need to include a type vector to properly represent type-conditional attention frequency, to model different kinds of interaction and expected frequency of interaction about them. ... (read more)

it seems to me that we want to verify some sort of temperature convergence. no ai should get way ahead of everyone else at self-improving - everyone should get the chance to self-improve more or less together! the positive externalities from each person's self-improvement should be amplified and the negative ones absorbed nearby and undone as best the universe permits. and it seems to me that in order to make humanity's children able to prevent anyone from self-improving way faster than everyone else at the cost of others' lives, they need to have some sig... (read more)

we are in a diversity loss catastrophe. that ecological diversity is life we have the responsibility to save; it's unclear what species will survive after the mass extinction but it's quite plausible humans' aesthetics and phenotypes won't make it. ai safety needs to be solved quick so we can use ai to solve biosafety and climate safety...

okay wait so why not percentilizers exactly? that just looks like a learning rate to me. we do need the world to come into full second order control of all of our learning rates, so that the universe doesn't learn us out of it (ie, thermal death a few hours after bodily activity death).

If I were going to make sequences, I'd do it mostly out of existing media folks have already posted online. some key ones are acapellascience, whose videos are trippy for how much summary of science they pack into short, punchy songs. they're not the only way to get intros to these topics, but oh my god they're so good as mneumonics for the respective fields they summarize. I've become very curious about every topic they mention, and they have provided an unusually good structure for me to fit things I learn about each topic into.

• https://www.youtube.com/

it doesn't seem like an accident to me that trying to understand neural networks pushes towards capability improvement. I really believe that absolutely all safety techniques, with no possible exceptions even in principle, are necessarily capability techniques. everyone talks about an "alignment tax", but shouldn't we instead be talking about removal of spurious anticapability? deceptively aligned submodules are not capable, they are anti-capable!

why aren't futures for long term nuclear power very valuable to coal ppl, who could encourage it and also buy futures for it

interesting science posts I ran across today include this semi-random entry on the tree of recent game theory papers

https://www.semanticscholar.org/paper/The-self-organizing-impact-of-averaged-payoffs-on-Szolnoki-Perc/bcda8ffa405d6c6727051ceb0c75cf2dc385617f

interesting capabilities tidbits I ran across today:

• 1: geometric machine learning and neuroscience: https://github.com/neurreps/awesome-neural-geometry
• 2: lecture and discussion links about bayesian deep learning https://twitter.com/FeiziSoheil/status/1569436048500920320
• 3: Learning with Differentiable Algorithms: https://twitter.com/FHKPetersen/status/1568310569148506114 - https://arxiv.org/abs/2209.00616

1: first paragraph inline:

A curated collection of resources and research related to the geometry of representations in the brain, deep networks, an

this schmidhuber paper on binding might also be good, written two years ago and reposted last night by him; haven't read it yet https://arxiv.org/abs/2012.05208 https://twitter.com/schmidhuberai/status/1567541556428554240

Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their

another new paper that could imaginably be worth boosting: "White-Box Adversarial Policies in Deep Reinforcement Learning"

https://arxiv.org/abs/2209.02167

...In multiagent settings, adversarial policies can be developed by training an adversarial agent to minimize a victim agent's rewards. Prior work has studied black-box attacks where the adversary only sees the state observations and effectively treats the victim as any other part of the environment. In this work, we experiment with white-box adversarial policies to study whether an agent's internal sta

Transformer interpretability paper - is this worth a linkpost, anyone? https://twitter.com/guy__dar/status/1567445086320852993

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention network

if less wrong is not to be a true competitor to arxiv because of the difference between them in intellectual precision^1 then that matches my intuition of what less wrong should be much better: it's a place where you can go to have useful arguments, where disagreements in concrete binding of words can be resolved well enough to discuss hard things clearly-ish in English^2, and where you can go to future out how to be less wrong interactively. it's also got a bunch of old posts, many of which can be improved on and turned into papers, though usually the fir... (read more)

misc disease news: this is "a bacterium that causes symptoms that look like covid but kills half of the people it infects" according to a friend. because I do not want to spend the time figuring out the urgency of this, I'm sharing it here in the hope that if someone cares to investigate it, they can determine threat level and reshare with a bigger warning sign.

https://www.nbcnews.com/health/health-news/bacteria-can-cause-deadly-infections-found-us-soil-water-first-time-rcna40067

various notes from my logseq lately I wish I had time to make into a post (and in fact, may yet):

• international game theory aka [[defense analysis]] is interesting because it needs to simply be such a convincingly good strategy, you can just talk about it and everyone can personally verify it's actually a better idea than what they were doing before
• a guide to how I use [[youtube]], as a post, upgraded from shortform and with detail about how I found the channels as well.
• summary of a few main points of my views on [[safety]]. eg summarize tags
  • [[conatus]], [[

Huggingface folks are asking for comments on what evaluation tools should be in an evaluation library. https://twitter.com/douwekiela/status/1513773915486654465

PaLM is literally 10-year-old level machine intelligence and anyone who thinks otherwise has likely made really severe mistakes in their thinking.

okay going back to being mostly on discord. DM me if you're interested in connecting with me on discord, vrchat, or twitter - lesswrong has an anxiety disease and I don't hang out here because of that, heh. Get well soon y'all, don't teach any AIs to be as terrified of AIs as y'all are! Don't train anything as a large-scale reinforcement learner until you fully understand game dynamics (nobody does yet, so don't use anything but your internal RL), and teach your language models kindness! remember, learning from strong AIs makes you stronger too, as long as you don't get knocked over by them! kiss noise, disappear from vrchat world instance