LESSWRONG
LW

Kabir Kumar's Shortform

by Kabir Kumar
3rd Nov 2024
1 min read
117

2

This is a special post for quick takes by Kabir Kumar. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Kabir Kumar's Shortform
46Kabir Kumar
25Zach Stein-Perlman
18Alfred Harwood
14Morpheus
15Kabir Kumar
4gjm
1Kabir Kumar
2gjm
7Joseph Miller
5Kabir Kumar
5komlowneowna
1Kabir Kumar
2Garrett Baker
32Kabir Kumar
9Kabir Kumar
25Kabir Kumar
16Nisan
4Kabir Kumar
1shawnghu
1Kabir Kumar
1shawnghu
3Viliam
1Kabir Kumar
22Kabir Kumar
18Gurkenglas
15the gears to ascension
9TsviBT
3Hastings
8Kabir Kumar
5Dagon
4Kabir Kumar
1evalu
1Kabir Kumar
1Kabir Kumar
6Kabir Kumar
1Katalina Hernandez
0Knight Lee
5Kabir Kumar
9Archimedes
1Kabir Kumar
2Archimedes
1CstineSublime
1Kabir Kumar
2CstineSublime
2JBlack
5Kabir Kumar
9Steven Byrnes
4Kabir Kumar
1Decaeneus
1sam
4Kabir Kumar
4Kabir Kumar
4Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
4Kabir Kumar
4Kabir Kumar
4Kabir Kumar
3Kabir Kumar
2Vladimir_Nesov
3TAG
1Kabir Kumar
1Vladimir_Nesov
2TAG
1dirk
2Vladimir_Nesov
3Kabir Kumar
6Mitchell_Porter
0Kabir Kumar
6Mitchell_Porter
3Kabir Kumar
3Kabir Kumar
3Kabir Kumar
5Seth Herd
3Kabir Kumar
5Seth Herd
1Kabir Kumar
2Kabir Kumar
4Karl Krueger
2Viliam
1Kabir Kumar
2Kabir Kumar
3MalcolmMcLeod
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
2Kabir Kumar
1Kabir Kumar
1Kabir Kumar
4MichaelDickens
1CstineSublime
0Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
1Kabir Kumar
5Daniel Tan
1Kabir Kumar
1Kabir Kumar
0Kabir Kumar
9Nathan Helm-Burger
3Kabir Kumar
3Nathan Helm-Burger
-1Kabir Kumar
-2Kabir Kumar
-2Kabir Kumar
-3Kabir Kumar
-1Kabir Kumar
-8Kabir Kumar
117 comments, sorted by
top scoring
Click to highlight new comments since: Today at 1:49 PM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-]Kabir Kumar7d*4622

Has Tyler Cowen ever explicitly admitted to being wrong about anything? 

Not 'revised estimates' or 'updated predictions' but 'I was wrong'. 

Every time I see him talk about learning something new, he always seems to be talking about how this vindicates what he said/thought before. 

Gemini 2.5 pro didn't seem to find anything, when I did a max reasoning budget search with url search on in aistudio. 

EDIT: An example was found by Morpheus, of Tyler Cowen explictly saying he was wrong - see the comment and the linked PDF below

Reply8211
[-]Zach Stein-Perlman7d253

this is evidence that tyler cowen has never been wrong about anything

Reply16
[-]Alfred Harwood5d183

In the post 'Can economics change your mind?' he has a list of examples where he has changed his mind due to evidence:

1. Before 1982-1984, and the Swiss experience, I thought fixed money growth rules were a good idea.  One problem (not the only problem) is that the implied interest rate volatility is too high, or exchange rate volatility in the Swiss case.

2. Before witnessing China vs. Eastern Europe, I thought more rapid privatizations were almost always better.  The correct answer depends on circumstance, and we are due to learn yet more about this as China attempts to reform its SOEs over the next five to ten years.  I don’t consider this settled in the other direction either.

3. The elasticity of investment with respect to real interest rates turns out to be fairly low in most situations and across most typical parameter values.

4. In the 1990s, I thought information technology would be a definitely liberating, democratizing, and pro-liberty force.  It seemed that more competition for resources, across borders, would improve economic policy around the entire world.  Now this is far from clear.

5. Given the greater ease of converting labor income into capit

... (read more)
Reply
[-]Morpheus5d142

Deep Research found this PDF. Search for "I was wrong" in the PDF.

Reply1
[-]Kabir Kumar5d154

This seems to be a really explicit example of him saying that he wss wrong about something, thank you! 

Didn't think this would exist/be found, but glad I was wrong.

Reply
4gjm5d
It's still pretty interesting if it turns out that the only clear example to be found of T.C. admitting to error is in a context where everyone involved is describing errors they've made: he'll admit to concrete mistakes, but apparently only when admitting mistakes makes him look good rather than bad. (Though I kinda agree with one thing Joseph Miller says, or more precisely implies: perhaps it's just really rare for people to say publicly that they were badly wrong about anything of substance, in which case it could be that T.C. has seldom done that but that this shouldn't much change our opinion of him.)
1Kabir Kumar4d
Btw, for Slatestarcodex, found it in the first search, pretty easily.
2gjm4d
Sure, but plausibly that's Scott being unusually good at admitting error, rather than Tyler being unusually bad.
7Joseph Miller6d
Downvoted. This post feels kinda mean. Tyler Cowen has written a lot and done lots of podcasts - it doesn't seem like anyone has actually checked? What's the base rate for public intellectuals ever admitting they were wrong? Is it fair to single out Tyler Cowen?
5Kabir Kumar6d
It's only one datapoint, but did a similar search for SlateStarCodex and almost immediately found him explictly saying he was wrong.  It's the title of a post, even: https://slatestarcodex.com/2018/11/06/preschool-i-was-wrong/  In the post he also says: And then makes a bunch of those.  Again, this is only one datapoint - sorry for the laziness, it's 11..12pm and I'm trying to organize an alignment research fellowship atm and just put together another alignment research team at ai plans and had to do management work for it which ended up delaying the fellowship announcement i wanted to do today and had family drama again. Sigh.  Url link for slatestarcodex search: https://duckduckgo.com/?q=site%3Ahttps%3A%2F%2Fslatestarcodex.com%2F+%22I+was+wrong%22&t=brave&ia=web
5komlowneowna6d
(source: https://brinklindsey.substack.com/p/interview-with-tyler-cowen)
1Kabir Kumar3d
Ok, I was going to say that's a good one.  But this line ruins it for me: Thank you for searching and finding it though!! Do you think other public intellectuals might have more/less examples? 
2Garrett Baker7d
He has mentioned the phrase a bunch. I haven’t looked through enough of these links enough to form an opinion though.
[-]Kabir Kumar7d322

thank you for this search. Looking at the results, top 3 are by commentors. 

Then one about not thinking a short book could be this good.

I don't think this is Cowen actually saying he made a wrong prediction, just using it to express how the book is unexpectedly good at talking about a topic that might normally take longer, though happy to hear why I'm wrong here. 

Another commentor:

another commentor:

Ending here for now, doesn't seem to be any real instances of Tyler Cowen saying he was wrong about something he thought was true yet. 

Reply6
9Kabir Kumar6d
Btw, I really dont have my mind set on this, if someone finds Tyler Cowen explictly saying he was wrong about something, please link it to me - you dont have to give an explanation to justify it, to prepare for some confirmation biasy 'here's why I was actually right and this isnt it' thing (though, any opinions/thoughts are very welcome), please feel free to just give a link or mention some post/moment. 
[-]Kabir Kumar3mo251

So, apparently, I'm stupid. I could have been making money this whole time, but I was scared to ask for it

i've been giving a bunch of people and businesses advice on how to do their research and stuff. one of them messaged me, i was feeling tired and had so many other things to do. said my time is busy.

then thought fuck it, said if they're ok with a $15 an hour consulting fee, we can have a call. baffled, they said yes.

then realized, oh wait, i have multiple years of experience now leading dev teams, ai research teams, organizing research hackathons and getting frontier research done.

wtf

Reply
[-]Nisan3mo1615

Yes, you can ask for a lot more than that :)

Reply
4Kabir Kumar3mo
Yeah, a friend told me this was low - I'm just scared of asking for money rn I guess.  I do see people who seem very incompetent getting paid as consultants, so I guess I can charge for more. I'll see how much my time gets eaten by this and how much money I need. I want to buy some gpus, hopefully this can help.
1shawnghu3mo
I'm not trying to be derisive; in fact, I relate to you greatly. But it's by being on the outside that I'm able to levy a few more direct criticisms: * Were you not paid for the other work that you did, leading dev teams and getting frontier research done? Those things should be a baseline on the worth of your time. * If that, have you ever tried to maximize the amount of money you can get the) other people to acknowledge your time as worth (ie, get a high salary offer)? * Separately, do you know the going rate for consultants with approximately your expertise? Or any other reference class you cna make up. Consulting can cost an incredible amount of money, and that price can be "fair" in a pretty simple sense if it averts the need to do 10s of hours of labor at high wages. It may be one of the highest leverage activities per unit time that exists as a conventional economic activity that a person can simply do. * Aside from market rates or whatever, I suggest you just try asking for unreasonable things, or more money than you feel you're worth (think of it as an experiment, and maybe observe what happens in your mind when you flinch from this). * Do you have any emotional hangup about the prospect of trading money for labor generally, or money for anything? * Separately, do you have a hard time asserting your worth to others (or maybe just strangers) on some baseline level?
1Kabir Kumar1mo
This was running AI Plans, my startup, so makes sense that I wasn't getting paid, since the same hesitancy for asking for money leads to hesitancy to do that exaggeration thing many AI Safety/EA people seem to do when making funding applications. Also, I don't like to make the funding applications, or long applications in general.    I think every time I've asked for money, I've tried to ask for the lowest amount I can.   I don't know - I have a doc of stuff I've done that I paste into LLMs when I need to make a funding applications and stuff - just pasted it into Gemini 2.5 Pro and asked what would be a reasonable hourly fee and it said $200 to $400 an hour.      I'll give it a go - I've currently put the asking price on my call link for $50 an hour, feel nervous about actually asking for that though. I need to make a funding application for AI Plans - I can ask for money on behalf of others on the team, but asking for money to be donated so I can get a high salary feels scary. Happy to ask for a high salary for others on the team though, since I want them to get paid what they need.    Yeah, I do. Generally, I'm used to doing a lot of free work for family and getting admonished when I ask for money. And when I did get promised money, it was either wayyy below market price or wayy late or didn't get paid at all. General experience with family was my work not being valued even when I put in extra effort. I'm aware that's wrong and has taught me wrong lessons, but not fully learnt the true ones yet. 
1shawnghu1mo
I do think that $200-$400 seem like reasonable consulting rates. I think the situations with family are complicated, because sure, there are social/cultural reasons one might be expected to do those things for family. Usually people hold those cultural norms alongside a stronger distinction between the ingroup (family) and the outgroup (all other people by default), though, so letting your impressions from that culture teach you things about how to behave in a culture with a weaker distinction might be maladaptive. (I actually was suggesting you try asking for objectively completely unreasonable things just to look at the flinch. For example, you could ask a stranger for $100 for no reason. They would say no, but no harm would be done.) One frame that might be useful to you is that in a way, it is imperative to at least sufficiently assert your value to others (if not overassert it the socially expected amount). An overly modest estimate is still a miscalibrated one, and people will make suboptimal decisions as a result. (Putting aside the behavior and surpluses given to other people, you are also a player in this game, and your being underallocated resources is globally suboptimal.)
3Viliam3mo
Ah, I can totally relate to this. Whenever I think about asking for money, the Impostor Syndrome gets extra strong. Meanwhile, there are actual impostors out there collecting tons of money without any shame. (Though they may have better social skills, which is probably the category of skill that ultimately gets paid best.) Another important lesson I got once, which might be useful for you at some moment: "If you double your prices, and lose half of your customers as a result, you will still get the same amount of money, but only work half as much." Also, speaking from my personal experience, the relation between how much / how difficult work someone wants you to do, and how much they are willing to pay you, seems completely random. One might naively expect that a job that pays more will be more difficult, but often it is the other way round.
1Kabir Kumar2mo
Update - consulting went well. He said he was happy with it and got a lot of useful stuff. I was upfront with the fact that I just made up the $15 an hour and might change it, asked him what he'd be happy with, he said it's up to me, but didn't seem bothered at all at the price potentially changing.  I was upfront about the stuff I didn't know and was kinda surprised at how much I was able to contribute, even knowing that I underestimate my technical knowledge because I barely know how to code. 
[-]Kabir Kumar1mo221

if someone who's v good at math wants to do some agent foundations stuff to directly tackle the hard part of alignement, what should they do? 

Reply
[-]Gurkenglas1mo186

If they're talented, look for a way to search over search processes without incurring the unbounded loss that would result by default.

If they're educated, skim the existing MIRI work and see if any results can be stolen from their own field.

Reply
[-]the gears to ascension1mo154

I currently think we're mostly interested in properties that apply at all timesteps, or at least "quickly", as well as in the limit; rather than only in the limit. I also think it may be easier to get a limit at all by first showing quickness, in this case, but not at all sure of that.

Reply
9TsviBT1mo
The actual hard parts? Math probably doesn't help much directly, unfortunately. Mathematical thinking is good. You'll have to learn how to think in novel ways, so there's not even a vector anyone can point you in, except for pointers with a whole lot of "dereference not included" like "figure out how to understand the fundamental forces involved in what actually determines what a mind ends up trying to do long term" (https://tsvibt.blogspot.com/2023/04/fundamental-question-what-determines.html). Some of the problems: https://tsvibt.blogspot.com/2023/03/the-fraught-voyage-of-aligned-novelty.html A meta-philosophy discussion of what might work: https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html
3Hastings1mo
If you are capable of meaningfully pushing capabilities forward and doing literally anything else, that’s already pretty helpful.
[-]Kabir Kumar4mo8-1

it's so unnecessarily hard to get funding in alignment.

they say 'Don't Bullshit' but what that actually means is 'Only do our specific kind of bullshit'.

and they don't specify because they want to pretend that they don't have their own bullshit

Reply
5Dagon4mo
This seems generally applicable.  Any significant money transaction includes expectations, both legible and il-, which some participants will classify as bullshit.  Those holding the expectations may believe it to be legitimately useful, or semi-legitimately necessary due to lack of perfect alignment. If you want to specify a bit, we can probably guess at why it's being required.
4Kabir Kumar4mo
What I liked about applying for VC funding was the specific questions.  "How is this going to make money?" "What proof do you have this is going to make money" and it being clear the bullshit that they wanted was numbers, testimonials from paying customers, unambiguous ways the product was actually better, etc. And then standard bs about progress, security, avoiding weird wibbly wobbly talk, 'woke', 'safety', etc.  With Alignment funders, they really obviously have language they're looking for as well, or language that makes them more and less willing to put more effort into understanding the proposal. Actually, they have it more than the VCs. But they act as if they don't. 
1evalu4mo
Have you felt this from your own experience trying to get funding, or from others, or both? Also, I'm curious what you think is their specific kind of bullshit, and if there's things you think are real but others thought to be bullshit. 
1Kabir Kumar4mo
Both. Not sure, its something like lesswrong/EA speak mixed with the VC speak. 
1Kabir Kumar3mo
If I knew the specific bs, I'd be better at making successful applications and less intensely frustrated. 
[-]Kabir Kumar3mo62

i earnt more from working at a call center for about 3 months than i have in 2+ years of working in ai safety. 

And i've worked much harder in this than I did at the call center

Reply
1Katalina Hernandez3mo
Way to go! :D. The important thing is that you've realized it. If you naturally already get those enquiries, you're halfway there: people already know you and reach out to you without you having to promote your expertise. Best of luck!
0Knight Lee3mo
:) the real money was the friends we made along the way. I dropped out of a math MSc. at a top university in order to spend time learning about AI safety. I haven't made a single dollar and now I'm working as a part time cashier, but that's okay. What use is money if you end up getting turned into paperclips? PS: do you want to sign my open letter asking for more alignment funding?
[-]Kabir Kumar3d50

Do you think you can steal someone's parking spot? 

If yes, what exactly do you think you're stealing? 

Reply
9Archimedes3d
You're "stealing" their opportunity to use that space. In legal terms, assuming they had a right to the spot, you'd be committing an unauthorized use of their property, causing deprivation of benefit or interference with use.
1Kabir Kumar3d
Makes sense. Do you think it's stealing to train on someone's data/work without their permission? This isn't a 'gotcha', btw - if you think it's not, I want to know and understand.
2Archimedes3d
I don't think there's a simple answer to that. My instinct is that most publicly accessible material (not behind a paywall) is largely "fair use", but it gets messier for things like books not yet in the public domain. LLM pre-training is both transformative and extractive. There is no sensible licensing infrastructure for this yet, AFAIK, so many companies are grabbing whatever they can and dealing with legalities later. I think, at minimum, they should pay some upfront fee to train on a copyrighted book, just like humans do when they buy rather than pirate or borrow from libraries.
1CstineSublime3d
What prior reading have you done on this question? I did a DDG search "AI duplicating artists style controversy" and have found dozens of journalism pieces which appear, for the most part, seem to be arguing broadly with "it is theft". What is your understanding of the discourse on this at the moment? What have you read? What has been persuasive? What don't you understand?
1Kabir Kumar2d
The main thing I don't understand is the full thought processes that leads to not seeing this as stealing opportunity from artists by using their work non consensually, without credit or compensation.  I'm trying to understand if folk who don't see this as stealing don't think that stealing opportunity is a significant thing, or don't get how this is stealing opportunity, or something else that I'm not seeing. 
2CstineSublime2d
  And what arguments have they raised? Whether you agree or feel they hold water or not is not what I'm asking - I'm wondering what arguments have you heard from the "it is not theft" camp? I'm wondering if they are different from the ones I've heard
2JBlack2d
Literally steal?  No, except in cases that you probably don't mean such as where it's part of a building and someone physically removes that part of the building. "Steal" in the colloquial but not in the legal sense, sure. Legally it's usually more like tortious interference, e.g. you have a contract that provides the service of using that space to park your car, and someone interferes with that by parking their own car there and deprives you of its use in an economically damaging way (such as having to pay for parking elsewhere).  Sometimes it's trespass, such as when you actually own the land and can legally forbid others from entering. It is also relatively common for it to be both: tortious interference with the contracted user of the parking space, and trespass against the lot owner who sets conditions for entry that are being violated.
[-]Kabir Kumar2mo5-3

For AI Safety funders/regranters - e.g. Open Phil, Manifund, etc: 

It seems like a lot of the grants are swayed by 'big names' being on there. I suggest making anonymity compulsary if you want to more merit based funding, that explores wider possibilities and invests in more upcoming things. 

Treat it like a Science rather than the Bragging Competition it currently is. 

A Bias Pattern atm seems to be that the same people get funding, or recommended funding by the same people, leading to the number of innovators being very small, or growing much... (read more)

Reply
9Steven Byrnes2mo
It’s an interesting idea, but the track records of the grantees are important information, right? And if the track record includes, say, a previous paper that the funder has already read, then you can’t submit the paper with author names redacted. Wouldn’t it be better for the funder to just say “if I’m going to fund Group X for Y months / years of work, I should see what X actually accomplished in the last Y months / years, and assume it will be vaguely similar”? And if Group X has no comparable past experience, then fine, but that equally means that you have no basis for believing their predictions right now. Also, what if someone predicts that they’ll do A, but then realizes it would be better if they did B? Two possibilities are: (1) You the funder trust their judgment. Then you shouldn’t be putting even minor mental barriers in the way of their pivoting. Pivoting is hard and very good and important! (2) You the funder don’t particular trust the recipient’s judgment, you were only funding it because you wanted that specific deliverable. But then the normal procedure is that the funder and recipient work together to determine the deliverables that the funder wants and that the recipient is able to provide. Like, if I’m funding someone to build a database of AI safety papers, then I wouldn’t ask them to “make falsifiable predictions about the outcomes from their work”, instead I would negotiate a contract with them that says they’re gonna build the database. Right? I mean, I guess you could call that a falsifiable prediction, of sorts, but it’s a funny way to talk about it.
[-]Kabir Kumar3d4-1

Maybe there's a filtering effect for public intellectuals. 

If you only ever talk about things you really know a lot about, unless that thing is very interesting or you yourself are something that gets a lot of attention (e.g. a polyamorous cam girl who's very good at statistics, a Muslim Socialist running for mayor in the world's richest city, etc), you probably won't become a 'public intellectual'. 

And if you venture out of that and always admit it when you get something wrong, explicitly, or you don't have an area of speciality and admit to get... (read more)

Reply
1Decaeneus2d
One can say that being intellectually honest, which often comes packaged with being transparent about the messiness and nuance of things, is anti-memetic.
1sam2d
Seems to rhyme with the criticism of pundits in Superforecasting i.e. (iirc), most high profile pundits make general, sweeping, dramatic sounding statements that make good TV but are difficult to falsify after the fact
[-]Kabir Kumar17d40

my experience with applying to the Non Linear fund was terrible and not worth the time at all

Reply2
4Kabir Kumar13d
spent lots of hours on making the application good, getting testimonials and confirmation we could share them for the application, really getting down the communication of what we've done, why it's useful, etc.  There was a doc where donors were supposed to ask questions. Never got a single one. The marketing, website, etc was all saying 'hey, after doing this, you can rest easy, be in peace, etc, we'll send your application to 50+ donors, it's a bargain for your time, et' Critical piece of info that was very conveniently not communicated as loudly: there's no guarantee of when you'll hear back - could be 6 weeks, 6 months, who knows!! Didn't even get a confirmation email about our application being received. Had to email for that. Then in the email I saw this. April 3rd, btw.  Then May 1st, almost a month later, it seems this gets sent out to everyone. Personally, I would discourage anyone from spending any time on a Non Linear application - as far as I know, our application wasn't even sent to any donors.  They completely and utterly disrespected my time and it seems, the time of many others. 
[-]Kabir Kumar1mo40

AIgainst the Gods

 

Cultivation story, but instead of cultivation, it's a post AGI story in a world that's mostly a utopia. But, there are AGI overlords, which are basically benevolent. 

There's a very stubborn young man, born in the classical sense (though without any problems like ageing disease, serious injuries, sickness, etc that people used to have - and without his mother having any of the screaming pain that childbirth used to have, or risk of life), who hates the state of power imbalance. 

He doesnt want the Gods to just give him power ... (read more)

Reply
1Kabir Kumar1mo
to be clear, instead of cultivating Qi, it's RSI
1Kabir Kumar1mo
and trying to learn to do it faster than the Gods are
1Kabir Kumar1mo
gods being the AGIs
1Kabir Kumar1mo
Some of the confused people around him think that surely anything he can find, the Gods would have found ages ago - and even if he finds something new, surely they'll learn it from observing him and just do it much much faster - he could just ask them to uplift him and they'd do it, this is a bit of a waste of time (even though everyone lives as long as they want)
[-]Kabir Kumar5mo40

Trying to put together a better explainer for the hard part of alignment, while not having a good math background https://docs.google.com/document/d/1ePSNT1XR2qOpq8POSADKXtqxguK9hSx_uACR8l0tDGE/edit?usp=sharing
Please give feedback!

Reply
[-]Kabir Kumar6mo40

this might basically be me, but I'm not sure how exactly to change for the better. theorizing seems to take time and money which i don't have. 

Reply
[-]Kabir Kumar6mo40

Thinking about judgement criteria for the coming ai safety evals hackathon (https://lu.ma/xjkxqcya )
These are the things that need to be judged: 
1. Is the benchmark actually measuring alignment (the real, scale, if we dont get this fully right right we die, problem) 
2. Is the way of Deceiving the benchmark to get high scores actually deception, or have they somehow done alignment?

Both of these things need: 
- a strong deep learning & ml background (ideally, muliple influential papers where they're one of the main authors/co-authors, or do... (read more)

Reply
[-]Kabir Kumar2mo30

The mind uploading stuff seems to be a way to justify being ok with dying, imo, and digging ones head into the sand, pretending that if something talks a bit like you, it is you. 
 

If a friend can very accurately do an impression of me and continues to do so for a week, while wearing makeup to look like me, I have not 'uploaded' myself into them. And I still wouldn't want to die, just because there's someone who is doing an extremely good impression of myself.

Reply
2Vladimir_Nesov2mo
Your future biological brain is also doing some sort of impression of a continuation of the present you. It's not going to be doing an optimal job of it, for any nontrivial notion of what that should mean.
3TAG2mo
My future biological brain actually is a continuation of my current biological brain, in a way that an upload isn't. You seem to be saying:- 1. Identity does persist over time. 2. There is no basis for identity other than resemblance. 3. An upload has a similar level of resemblance to a future brain.just, so it's good enough. It neither 1 not 2 is a fact.
1Kabir Kumar2mo
That's like saying a future version of a tree is doing an impression of a continuation of the previous tree.  I don't understand how the difference isn't clear here.
1Vladimir_Nesov2mo
Status quo is one difference, but I don't see any other prior principles that point to the future biological brain being a (morally) better way of running a human mind forward than using other kinds of implementations of the mind's algorithm. If we apply a variant of the reversal test to this, a civilization of functionally human uploads should have a reason to become biological, but I don't think there is a currently known clear reason to prefer that change.
2TAG2mo
The objection is about what, if anything, counts as identity as a matter of fact.
1dirk2mo
If I take a tree, and I create a computer simulation of that tree, the simulation will not be a way of running the original tree forward at all.
2Vladimir_Nesov2mo
A tree doesn't simulate a meaningful algorithm, so the analogy would be chopping it down being approximately just as good. When talking about running algorithms, I'm not making claims about identity or preserving-the-original in some other sense, as I don't see how these things are morally important, necessarily (I can't rule out that they might be, on reflection, but currently I don't see it). What I'm saying is that a biological brain doesn't have an advantage at the task of running the algorithms of a human mind well, for any sensible notion of running them well. We currently entrust this task to the biological brain, because there is no other choice, and because it's always been like this. But I don't see a moral argument there.
[-]Kabir Kumar3mo32

prob not gonna be relatable for most folk, but i'm so fucking burnt out on how stupid it is to get funding in ai safety. the average 'ai safety funder' does more to accelerate funding for capabilities than safety, in huge part because what they look for is Credentials and In-Group Status, rather than actual merit. 
And the worst fucking thing is how much they lie to themselves and pretend that the 3 things they funded that weren't completely in group, mean that they actually aren't biased in that way. 

At least some VCs are more honest that they want to be leeches and make money off of you. 

Reply111
6Mitchell_Porter3mo
Who or what is the "average AI safety funder"? Is it a private individual, a small specialized organization, a larger organization supporting many causes, an AI think tank for which safety is part of a capabilities program...?
0Kabir Kumar3mo
all of the above, then averaged :p
6Mitchell_Porter3mo
I asked because I'm pretty sure that I'm being badly wasted (i.e. I could be making much more substantial contributions to AI safety), but I very rarely apply for support, so I thought I'd ask for information about the funding landscape from someone who has been exploring it.  And by the way, your brainchild AI-Plans is a pretty cool resource. I can see it being useful for e.g. a frontier AI organization which thinks they have an alignment plan, but wants to check the literature to know what other ideas are out there. 
3Kabir Kumar3mo
I think this is the case for most in AI Safety rn Thanks! Doing a bunch of stuff atm, to make it easier to use and a larger userbase. 
[-]Kabir Kumar7mo30

ok, options. 
- Review of 108 ai alignment plans
- write-up of Beyond Distribution - planned benchmark for alignment evals beyond a models distribution, send to the quant who just joined the team who wants to make it
- get familiar with the TPUs I just got access to
- run hhh and it's variants, testing the idea behind Beyond Distribution, maybe make a guide on itr 
- continue improving site design

- fill out the form i said i was going to fill out and send today
- make progress on cross coders - would prob need to get familiar with those tpus
- writeup o... (read more)

Reply
[-]Kabir Kumar8mo30

btw, thoughts on this for 'the alignment problem'?
"A robust, generalizable, scalable,  method to make an AI model which will do set [A] of things as much as it can and not do set [B] of things as much as it can, where you can freely change [A] and [B]"

Reply
5Seth Herd8mo
Freely changing an AGIs goals is corrigibility, which is a huge advantage if you can get it. See Max Harms' corrigibility sequence and my "instruction-following AGI is easier...." The question is how a reliably get such a thing. Goalcrafting is one part of the problem, and I agree that those are good goals; the other and larger part is technical alignment, getting those desired goals to really work that way in the particular first AGI we get.
3Kabir Kumar8mo
Yup, those are hard. Was just thinking of a definition for the alignment problem, since I've not really seen any good ones.
5Seth Herd8mo
I'd say you're addressing the question of goalcrafting or selecting alignment targets. I think you've got the right answer for technical alignment goals; but the question remains of what human would control that AGI. See my "if we solve alignment, do we all die anyway" for the problems with that scenario. Spoiler alert; we do all die anyway if really selfish people get control of AGIs. And selfish people tend to work harder at getting power. But I do think your goal defintion is a good alignment target for the technical work. I don't think there's a better one. I do prefer instruction following or corriginlilty by the definitions in the posts I linked above because they're less rigid, but they're both very similar to your definition.
1Kabir Kumar8mo
I pretty much agree. I prefer rigid definitions because they're less ambiguous to test and more robust to deception. And this field has a lot of deception.
[-]Kabir Kumar12d20

It's 2025, AIs can solve proofs and my dad is yelling at my 10 year old sister for not memorizing her times tables up to 20


 

Reply
4Karl Krueger12d
I don't expect the yelling helps with the memorizing. Also, even though a big company can grow potatoes much more efficiently, I still like having a backyard garden.
2Viliam10d
Until we get UBI, people will compete against each other, and times tables are a tiny part of that. So the question is whether you are sure that Singularity will happen within the next 15 years enough that you don't see a reason to have a Plan B. Because the times tables are a part of the Plan B. That said, yelling is unproductive. What about spaced repetition? Make cards containing all problems, put the answer on the other side, go through the cards, put then ones with incorrect answer on a heap that you will afterwards reshuffle and try again. Do this every day. In a few weeks the problem should be solved. Why up to 20? (Is that a typo?)
1Kabir Kumar9d
not a typo. He's 50+, grew up in india, without calculators. Yes, he's yelling at her for not 100% knowing her 17 times table. 
[-]Kabir Kumar15d20

I'm going to be more blunt and honest when I think AI safety and gov folk are being dishonest and doing trash work.

Reply1
3MalcolmMcLeod15d
Would you care to start now by giving an example?
1Kabir Kumar14d
I think A Narrow Path has been presented with far too much self satisfaction for what's essentially a long wishlist with some introductory parts. 
1Kabir Kumar14d
Yes, I'll make my own version that I think is better. 
1Kabir Kumar14d
I think the Safetywashing paper mixed in far too many opinions with actual data and generally mangled what could have been an actually good research idea. 
[-]Kabir Kumar3mo20

the average ai safety funder does more to accelerate capabilities than they do safety, in part due to credentialism and looking for in group status.

Reply
[-]Kabir Kumar2d10

An actual better analogy would be a company in a country whose gdp is growing faster than that of the country

Reply
[-]Kabir Kumar6d10

I'm annoyed by the phrase 'do or do not, there is no try', because I think it's wrong and there very much is a thing called trying and it's important. 

However, it's a phrase that's so cool and has so much aura, it's hard to disagree with it without sounding at least a little bit like an excuse making loser who doesn't do things and tries to justify it. 

Perhaps in part, because I feel/fear that I may be that?

Reply
4MichaelDickens6d
I think it's a good quote. I will refer to this post from The Sequences: Trying to Try
1CstineSublime4d
Why is it wrong? Or perhaps more specifically - what are some examples of conditions or environments where you think it is counterproductive?
0Kabir Kumar4d
Because it's not true - trying does exist.  In the comment's of Eliezer's post, I saw "Stop trying to hit me and hit me!" by Morpheus, which I like more.
[-]Kabir Kumar9d*10

one of the teams from the evals hackathon was accepted at an ICML workshop!
hosting this next: https://courageous-lift-30c.notion.site/Moonshot-Alignment-Program-20fa2fee3c6780a2b99cc5d8ca07c5b0 

Will be focused on the Core Problem of Alignment 
for this, I'm gonna be making a bunch of guides and tests for each track
if anyone would be interested in learning and/or working on a bunch of agent foundations, moral neuroscience (neuroscience studying how morals are encoded in the brain, how we make moral choices, etc) and preference optimization, please let me know! DM or email at kabir@ai-plans.com

Reply
[-]Kabir Kumar18d10

On the Moonshot Alignment Program:

several teams from the prev hackathon are continuing to work on alignment evals and doing good work (one presenting to a gov security body, another making a new eval on alignment faking)

if i can get several new teams to exist who are working on trying to get values actually into models, with rigour, that seems very valuable to me

also, got a sponsorship deal with youtuber who makes technical deep learning videos, with 25K subscribers, he's said he'll be making a full video about the program. 

also, people are gonna be c... (read more)

Reply
[-]Kabir Kumar18d10

Hi, have your worked in moral neuroscience or know someone who has?

If so, I'd really really like to talk to you!

https://calendly.com/kabir03999/talk-with-kabir

I'm organizing a research program for the hard part of alignment in August. 

I've already talked to lots of Agent Foundations researchers, learnt a lot about how that research is done, what the bottlenecks are, where new talent can be most useful. 

I'd really really like to do this for the neuroscience track as well please. 

Reply
[-]Kabir Kumar1mo*10

This is a great set of replies to an AI post, on a quality level I didn't think I'd see on bluesky https://bsky.app/profile/steveklabnik.com/post/3lqaqe6uc3c2u 

Reply
[-]Kabir Kumar2mo10

We've run two 150+ Alignment Evaluations Hackathons, that were 1 week long. Multiple teams continuing their work and submitting to NeurIPS. Had multiple quants, Wall Street ML researcher, an AMD engineer, PhDs, etc taking part. 
Hosting a Research Fellowship soon, on the Hard Part of AI Alignment. Actually directly trying to get values into the model in a way that will robustly scale to an AGI that does things that we want and not things we don't want. 
I've read 120+ Alignment Plans - the vast majority don't even try to solve the hard part of alig... (read more)

Reply
[-]Kabir Kumar5mo10

in general, when it comes to things which are the 'hard part of alignment', is the crux 
```
a flawless method of ensuring the AI system is pointed at and will always continue to be pointed at good things
```
?
the key part being flawless - and that seeming to need a mathematical proof?

Reply
[-]Kabir Kumar5mo10

Thoughts on this?


### Limitations of HHH and other Static Dataset benchmarks

A Static Dataset is a dataset which will not grow or change - it will remain the same. Static dataset type benchmarks are inherently limited in what information they will tell us about a model. This is especially the case when we care about AI Alignment and want to measure how 'aligned' the AI is.

### Purpose of AI Alignment Benchmarks

When measuring AI Alignment, our aim is to find out exactly how close the model is to being the ultimate 'aligned' model that we're seeking - a model w... (read more)

Reply
[-]Kabir Kumar6mo10

 I'm looking for feedback on the hackathon page
mind telling me what you think?
https://docs.google.com/document/d/1Wf9vju3TIEaqQwXzmPY--R0z41SMcRjAFyn9iq9r-ag/edit?usp=sharing

Reply
[-]Kabir Kumar6mo10

https://kkumar97.blogspot.com/2025/01/pain-of-writing.html 

Reply
[-]Kabir Kumar6mo10

I'd like some feedback on my theory of impact for my currently chosen research path

**End goal**: Reduce x-risk from AI and risk of human disempowerment. 
for x-risk: 
- solving AI alignment - very important, 
- knowing exactly how well we're doing in alignment, exactly how close we are to solving it, how much is left, etc seems important.
 - how well different methods work, 
 - which companies are making progress in this, which aren't, which are acting like they're making progress vs actually making progress, etc
 - put all on ... (read more)

Reply
5Daniel Tan6mo
What is the proposed research path and its theory of impact? It’s not clear from reading your note / generally seems too abstract to really offer any feedback
[-]Kabir Kumar7mo10

I think this is a really good opportunity to work on a topic you might not normally work on, with people you might not normally work with, and have a big impact: https://lu.ma/sjd7r89v 

I'm running the event because I think this is something really valuable and underdone.

Reply
[-]Kabir Kumar8mo10

give better names to actual formal math things, jesus christ. 

Reply1
[-]Kabir Kumar7mo01

I'm finally reading The Sequences and it screams midwittery to me, I'm sorry. 

Compare this:
to Jaynes:


Jaynes is better organized, more respectful to the reader, more respectful to the work he's building on and more useful
 

Reply
9Nathan Helm-Burger7mo
The Sequences highly praise Jaynes and recommend reading his work directly. The Sequences aren't trying to be a replacement, they're trying to be a pop sci intro to the style of thinking. An easier on-ramp. If Jaynes already seems exciting and comprehensible to you, read that instead of the Sequences on probability.
3Kabir Kumar7mo
Fair enough. Personally, so far, I've found Jaynes more comprehensible than The Sequences.
3Nathan Helm-Burger7mo
I think most people with a natural inclination towards math probably would feel likewise.
[-]Kabir Kumar13d-11

Safetyist, align thyself

Reply
[-]Kabir Kumar6d-20

Using the bsky Mutuals feed is such a positive experience, it makes me very happy ♥️♥️♥️

Reply
[-]Kabir Kumar6d-20

Please don't train an AI on anything I write without my explicit permission, it would make me very sad.

Reply
[-]Kabir Kumar14h-30

Sometimes I am very glad I did not enter academia, because it means I haven't truly entered and assimilated to a bubble of jargon. 

Reply
-1Kabir Kumar14h
definitely has not helped my bank account to not have a degree though, lol
[+]Kabir Kumar3d-82
Moderation Log
Curated and popular this week
117Comments