All of duck_master's Comments + Replies

You are allowed to edit Wikipedia

This is an extremely important point. (I remember thinking a long time ago that Wikipedia just Exists, and that although random people are allowed to edit it, doing it is generally Wrong.) FWIW I'm an editor now - User:Duckmather.

8ChristianKl24dThat's great to hear. Effects like this are what I was hoping for.
The Neglected Virtue of Scholarship

In fact, organized resources like Wikipedia, LW sequences, SEP, etc. are basically amortized scholarship. (This is particularly true for Wikipedia; its entire point is that we find vaguely-related content from around - or beyond - the web and then paraphrase it into a mildly-coherent article. Source: am wikipedia editor.)

Bad names make you open the box

I also agree that, for the purpose of previewing the content, this post is poorly titled (maybe it should be titled something like "Having bad names makes you open the black box of the name", except more concise?), although, for me, I didn't as much stick to a particular wrong interpretation as just view the entire title as unclear.

1Ericf2moSaying poor naming instead of bad names would be clearer, since it wouldn't call up the idea of "bad names" = swear words. Saying "look in" instead of "open" would also distance from the AI concept.
Problems facing a correspondence theory of knowledge

Thanks for the reply. I take it that not only are you interested in the idea of knowledge, but that you are particularly interested in the idea of actionable knowledge. 

Upon further reflection, I realize that all of the examples and partial definitions I gave in my earlier comment can in fact be summarized in a single, simple definition: a thing X has knowledge of a fact Y iff it contains some (sufficiently simple) representation of Y. (For example, a rock knows about the affairs of humans because it has a representation of those affairs in the form o... (read more)

2alexflint2moI very much agree with the emphasis on actionability. But what is it about a physical artifact that makes the knowledge it contains actionable? I don't think it can be simplicity alone. Suppose I record the trajectory of the moon over many nights by carving markings into a piece of wood. This is a very simple representation, but it does not contain actionable knowledge in the same way that a textbook on Newtonian mechanics does, even if the textbook were represented in a less simple way (say, as a PDF on a computer).
Problems facing a correspondence theory of knowledge

I think knowledge as a whole cannot be absent, but knowledge of a particular fact can definitely be absent (if there's no relationship between the thing-of-discourse and the fact).

1TAG2moSo rocks have non zero knowledge?
Predict responses to the "existential risk from AI" survey

Since this is a literally a question about soliciting predictions, it should have one of those embedded-interactive-predictions-with-histograms gadgets* to make predicting easier. Also, it might be worth it to have two prediction gadgets, since this is basically a prediction: one gadget to predict what Recognized AI Safety Experts (tm) predict about how much damage unsafe AIs will do, and one gadget to predict about how much damage unsafe AIs will actually do (to mitigate weird second-order effects having to do with predicting a prediction). 

*I'm not sure what they're supposed to be called.

2Rob Bensinger2moI think it might be more interesting to sketch what you expect the distribution of views to look like, as opposed to just giving a summary statistic. I can add probability Qs, but I avoided it initially so as not to funnel people into doing the less informative version of this exercise.
Problems facing a correspondence theory of knowledge

Au contraire, I think that "mutual information between the object and the environment" is basically the right definition of "knowledge", at least for knowledge about the world (as it correctly predicts that all four attempted "counterexamples" are in fact forms of knowledge), but that the knowledge of an object also depends on the level of abstraction of the object which you're considering.

For example, for your rock example: A rock, as a quantum object, is continually acquiring mutual information with the affairs of humans by the imprinting of subatomic in... (read more)

1TAG2moThen how can it ever be absent?
3alexflint2moThank you for this comment duck_master. I take your point that it is possible to extract knowledge about human affairs, and about many other things, from the quantum structure of a rock that has been orbiting the Earth. However, I am interested in a definition of knowledge that allows me to say what a given AI does or does not know, insofar as it has the capacity to act on this knowledge. For example, I would like to know whether my robot vacuum has acquired sophisticated knowledge of human psychology, since if it has, and I wasn't expecting it to, then I might choose to switch it off. On the other hand, if I merely discover that my AI has recorded some videos of humans then I am less concerned, even if these videos contain the basic data necessary to constructed sophisticated knowledge of human psychology, as in the case with the rock. Therefore I am interested not just in information, but something like action-readiness. I am referring to that which is both informative and action-ready as "knowledge", although this may be stretching the standard use of this term. Now you say that we might measure more abstract kinds of knowledge by looking at what an AI is willing to bet on. I agree that this is a good way to measure knowledge if it is available. However, if we are worried that an AI is deceiving us, then we may not be willing to trust its reports of its own epistemic state, or even of the bets it makes, since it may be willing to lose money now in order to convince us that it is not particularly intelligent, in order to make a treacherous turn later. Therefore I would very much like to find a definition that does not require me to interact with the AI through its input/output channels in order to find out what it knows, but rather allows me to look directly at its internals. I realize this may be impossible, but this is my goal. So as you can see, my attempt at a definition of knowledge is very much wrapped up with the specific problem I'm trying to solve, and
The secret of Wikipedia's success

I think this applies to every wiki ever, and also to this very site. There are probably a lot of others that I'm missing but this is a start.

The secret of Wikipedia's success

I agree with you (meaning G Gorden Worley III) that Wikipedia is reliable, and I too treat it as reliable. (It's so well-known as a reliable source that even Google uses it!) I also agree that an army of bots and humans undo any defacing that may occur, and that Wikipedia having to depend on other sources helps keep it unbiased. I also agree with the OP that Wikipedia's status as not-super-reliable among the Powers that Be does help somewhat.

So I think that the actual secret of Wikipedia's success is a combination of the two: Mild illegibility prevents ram... (read more)

2G Gordon Worley III3moOne bit of nuance my original comment leaves out is how flexible the citation policy is. Yes citations are required to include content on Wikipedia if it's not considered common knowledge, but also it's not that hard to produce something that Wikipedia can then cite, even if it must be referenced obliquely like "some people say X is true about Y". This is generally how Wikipedia deals with controversial topics today: cite sources expressing views in order to acknowledge the existence of disagreements and also keep disputed facts quarantined in "controversy" sections.
"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

@Diffractor: I think I got a MIRIxDiscord invite in a way somehow related to this event. Check your PMs for details. (I'm just commenting here to get attention because I think this might be mildly important.)

1duck_master3moBumping this.
"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

Don't worry, it was kind of a natural stopping point anyways, as the discussion was winding down.

2Ben Pace4moOh woops, I realize I ended the call for everyone when I left. I'm sorry.
"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

"Mixture of infra-distributions" as in convex set, or something else? If it's something else then I'm not sure how to think about it properly.

2Diffractor4mo"mixture of infradistributions" is just an infradistribution, much like how a mixture of probability distributions is a probability distribution. Let's say we've got a priorζ∈ΔN, a probability distribution over indexed hypotheses. If you're working in a vector space, you can take any countable collection of sets in said vector space, and mix them together according to a priorζ∈ΔNgiving a weight to each set. Just make the set of all points which can be made by the process "pick a point from each set, and mix the points together according to the probability distributionζ" For infradistributions as sets of probability distributions or a-measures or whatever, that's a subset of a vector space. So you have a bunch of setsΨi, and you just mix the sets together according toζ, that gives you your setΨζ. If you want to think about the mixture in the concave functional view, it's even nicer. You have a bunch ofψi:(X→R)→Rwhich are "hypothesis i can take a function and output what its worst-case expectation value is". The mixture of these,ψζ, is simply defined asψζ(f):=Ei∼ζ[ψi(f)]. This is just mixing the functions together! Both of these ways of thinking of mixtures of infradistributions are equivalent, and recover mixture of probability distributions as a special case.
"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

Me too. I currently only have a very superficial understanding of infraBayesianism (all of which revolves around the metaphysical, yet metaphorical, deity Murphy).

"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

More specifically: if two points are in a convex set, then the entire line segment connecting them must also be in the set.

"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

Here's an ELI5: The evil superintelligent deity Murphy, before you were ever conceived, picked the worst possible world that you could live in (meaning the world where your performance is worst), and you have to use fancy math tricks to deal with that.

"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

I think that if you imagine the deity Murphy trying to foil your plans whatever you do, that gives you a pretty decent approximation to true infraBayesianism.

"Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

Google doc where we posted our confusions/thoughts earlier: https://docs.google.com/document/d/1lKG_y_Voe02OkRGG9yaxtMuGM_dQBUKjj9DXTA8rMxE/edit

My ongoing confusions/thoughts:

  • What if the super intelligent deity is less than maximally evil or maximally good? (E.g. the deity picking the median-performance world)
  • What about the dutch-bookability of infraBayesians? (the classical dutch-book arguments seem to suggest pretty strongly that non-classical-Bayesians can be arbitrarily exploited for resources)
  • Is there a meaningful metaphysical interpretation of infraB
... (read more)
2Vanessa Kosoy4moThinking of the worst-case is just a mathematical reflection of the fact we want to be able to prove lower bounds on the expected utility of our agents. We have an unpublished theorem that, in some sense, any such lower bound guarantee has an infra-Bayesian formulation. Another way to justify it is the infra-Bayesian CCT (see "Complete Class Theorem Weak Version" here [https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa/p/e8qFDMzs2u9xf5ie6]). I think it might depend on the specific Dutch book argument, but one way infra-Bayesians escape them is by... being equivalent to certain Bayesians! For example, consider the setting where your agent has access to random bits that the environment can't predict. Then, infra-Bayesian behavior is just the Nash equilibrium in a two-player zero-sum game (against Murphy). Now, the Nash strategy in such a game is the (Bayes) optimal response to the Nash strategy of the other player, so it can be regarded as "Bayesian". However, the converse is false: not every best response to Nash is in itself Nash. So, the infra-Bayesian decision rule is more restrictive than the corresponding Bayesian decision rule, but it's a special case of the latter. I think of it as just another way of organizing uncertainty. The question is too broad for a succinct answer, I think, but here's one POV you could take: Let's remember the frequentist definition of probability distributions as time limits of frequencies. Now, what if the time limit doesn't converge? Then, we can make a (crisp) infradistribution instead: the convex hull of all limit points. Classical frequentism also has the problem that the exact same event never repeats itself. But in "infra-frequentism" we can solve this: you don't need the exact same event to repeat, you can draw the boundary around what counts as "the event" any way you like. Once we go from passive observation to active interaction with the environment, your own behavior serves as another source of Knightian uncertainty. That i