Joachim Bartosik

Wiki Contributions


I don't feel this way about something like, say, taking oral vitamin D in the winter. That's not in opposition to some adaptive subsystem in me or in the world. It's actually me adapting to my constraints.

If someone's relationship to caffeine were like that, I wouldn't say it's entropy-inducing.


I think this answers a question / request for clarification I had. So now I don't have to ask.

(The question was something like "But sometimes I use caffeine because I don't want to fall asleep while I'm driving (and things outside my controll made it so that doing a few hundred of driving km now-ish is the best option I can see)").

But in that case we just apply verification vs generation again. It's extremely hard to tell if code has a security problem, but in practice it's quite easy to verify a correct claim that code has a security problem. And that's what's relevant to AI delegation, since in fact we will be using AI systems to help oversee in this way.


I know you said that you're not going to respond but in case you feel like giving a clarification I'd like to point out that I'm confused here.

Yes it usually easy to verify that a specific problem exists if the exact problem is pointed out to you[1].

But it's much harder to verify claim that there are no problems, this code is doing exactly what you want.

And AFAIK staying in a loop:

1) AI tells us "here's a specific problem"

2) We fix the problem then 

3) Go back to step 1)

Doesn't help with anything? We want to be in a state where AI says "This is doing exactly what you want" and we have reasons to trust that (and that is hard to verify).

EDIT to add: I think I didn't make it clear enough what clarification I'm asking for. 

  • Do you think it's possible to use AI which will point out problems (but which we can't trust when it says everything is ok) to "win"? It would be very interesting if you did and I'd love to learn more.
  • Do you think that we could trust AI when it says that everything is ok? Again that'd be very interesting.
  • Did I miss something? I'm curious to learn what but that's just me being wrong (but that's not new path to win interesting).

Also it's possible that there are two problems, each problem is easy to fix on its own but it's really hard to fix them both at the same time (simple example: it's trivial to have 0 false positives or 0 false negatives when testing for a disease; it's much harder to eliminate both at the same time).

[1] Well it can be hard to reliably reproduce problem, even if you know exactly what the problem is (I know because I couldn't write e2e tests to verify some bug fixes).

What examples of practical engineering problems actually have a solution that is harder to verify than to generate? 


My intuition says that we're mostly engineering to avoid problems like that, because we can't solve them by engineering. Or use something other than engineering to ensure that problem is solved properly.

For example most websites don't allow users to enter plain html. Because while it's possible to write non-harmful html it's rather hard to verify that a given piece of html is indeed harmless. Instead sites allow something like markdown or visual editors which make it much easier to ensure that user-generated content is harmless. (that's example of engineering to avoid having to verify something that's very hard to verify)

Another example is that some people in fact can write html for those websites. In many places there is some process to try and verify they're not doing anything harmful. But those largely depend on non-engineering to work (you'll be fired and maybe sued if you do something harmful) and the parts that are engineering (like code reviews) can be fooled because they rely on assumption of your good intent to work (I think; I've never tried to put harmful code in any codebase I've worked with; I've read about people doing that).

I'm confused. What is the outer optimization target for human learning?

My two top guesses below.

To me it looks like human values are result of humans learning from environment (which was influenced by humans before and includes current humans). So it's kind of like human values are what humans learned by definition. So observing that humans learned human values doesn't tell us anything.

Or maybe you mean something like parents / society / ... teaching new humans their values? I see some other problems there:

  • I'm not sure what's success rate but values seem to be changing noticeably
  • There was a lot of time to test multiple methods of teaching new humans values, with humans not changing that much.

This doesn't always work: sometimes people develop an avoidance to going to the doctor or thinking about their health problems because of this sort of wireheading. 


Yes, but I'd like to understand how sometimes it does work.

I think I was thinking about this post. I'm still interested in learning where I could learn more about this (I now can try to backtrack from the post but since it links to a debate it might be hard to get to sources).

Yes, I felt that I was missing a point, thank you for pointing to the thing you found interesting in it.

it's easier to put yourself into the other person's ontology and get the message across in terms that they would understand, rather than trying to explain all of science.

Is a thing that makes sense. But I think the quote doesn't point at it very well. First a big chunk of it is asserting that belief in witchcraft theory of disease is similar to belief in germ theory of disease. (I don't know how well average person understands what are viruses)

Second where it talks about convincing by making concepts similar it's weird. For example

... people give untreated well water to their babies. The children regularly get diarrhea, and many of them die  (...) if they boil the water, it will kill these bacteria. A month later she’s back, and they’re still giving the babies the dirty water. After all, if a stranger came into your community and told you that your children got influenza because of witchcraft, would you respond by going out and slaughtering a sheep?

Influenza is much smaller risk than cholera (quick search says CFR for untreated cholera 25-50%, for flu 0.1%) and boiling water is much less costly than slaughtering sheep (it's likely to result in prison time where I live). (EDIT to add: I didn't check those numbers so don't trust them too much, they're just first numbers I could find and they roughly match my expectations)

Again thanks for explaining. (at least for me) your comment made the point much better than the quote in the post.

I'm not sure what you find interesting about the quote but I think it's pretty badly mistaken in trying to make it look like belief in witchcraft is very similar to belief in viruses.


When people get sick for unaccountable reasons in Manhattan, there is much talk of viruses and bacteria. Since doctors do not claim to be able to do much about most viruses, they do not put much effort into identifying them. Nor will the course of a viral infection be much changed by a visit to the doctor. In short, most appeals in everyday life to viruses are like most everyday appeals to witchcraft. They are supported only by a general conviction that sickness can be explained, and the conviction that viruses can make you sick.

Two things look wrong to me here:

  • It's pretty important to distinguish bacterial infections (for which there can be effective treatment) from viral infections (where I think often there is no treatment). (I'm pretty sure witchcraft theory of sickness can't tell the difference very well) (also vaccines are a things and work against some viruses, AFAIK witchcraft doesn't have anything similar)
  • If you suspect you have a viral infection and you care to avoid infecting others you can take an effective preventive measures. Not sure if witchcraft theory of sickness could help you.

If you ask most people in Manhattan why they believe in viruses, they will say two kinds of things: First, they will appeal to authority. “Science has shown,” they will say, though if you ask them how science showed it, you will pretty quickly reach an impasse

I'm not in Manhattan so I'm not sure if is counts. (I'm also not a virologist) But it should be relatively easy to design experiments to demonstrate that some diseases are transmitted by replicating things. First check if you can infect one subject with (a thing like a bit of snot) from a sick subject. Then try infect a bunch of subjects with diluted (thing like a bit of snot) to check how much you can lower the dose before it stops being infectious. Then take (a thing like a bit of snot) that received ~minumum infective dose, dilute it a bit (so if it was a non-replicating thing like poison dose would be too low to cause symptoms), infect some more subjects & you're done.

(actually running this would be probably pretty hard)

This gets you only that a replicating thing is causing the infection. To check that it's a virus... well you need a really good microscope so you can identify all the really small things, something to let you separate them and check which kind of the small thing is causing infection.

(also as far as I can tell this is asking much less from witchcraft; I'm not sure if witchcraft can tell difference between poisoning, bacterial infection, parasite infection, viral infection, vitamin deficiency, ....) (some of those are much easier to distinguish and treat than many viral infections)

I've seen the idea in this post:

Every now and then, you’ll have an opportunity to get great leverage on your money, your time, your energy, your friends, your internet connection, and so forth. Most days, the value of free time is relatively low, and can even be negative (“I’m bored!”), but when you need that time, you really need it. Money isn’t that important most of the time, but when you need it and don’t have it, it’s really bad. Most of the time being low energy, or not having as many friends as you’d like, or having a spotty internet connection, is mostly harmless, but at the wrong time it can spell disaster.

Using time now to save time later is often efficient even if you spend more time than you save, because that time later on might be orders of magnitude more important.

I think it's very important idea and one many people seem to not understand. For example back people complain that mandatory software updates disrupt their work. But usually you have some day to install any update, you could trigger update when you finish your work on first day it's available and never be forced to update when you don't want to.

you presumably don’t hand out any company credit cards at least outside of special circumstances.


This reminded me of an anecdote from "Surely You're Joking, Mr. Feynman!" where Feynman says that he

had been used to giving lectures for some company or university or for ordinary people, not for the government. I was used to, "What were your expenses?" ­­ "So­and­so much." ­­ "Here you are, Mr. Feynman.

I remember reading that and thinking that it's different from what I have to do (at a private company) when I want to expense something. I wonder if things were really done differently back then.  And how people made it work.

Load More