LessWrong has many criticisms of P-values/Statistical significance, And is often given as a coordination problem (or an inadequate equilibria).

In addition to the American Statistical Association - which, In 2016, 'released a statement in The American Statistician warning against the misuse of statistical significance and P values' - these authors now join the call too, with an article in nature signed by over 800 scientists. they write:

We are far from alone. When we invited others to read a draft of this comment and sign their names if they concurred with our message, 250 did so within the first 24 hours. A week later, we had more than 800 signatories.

What do you think will be the impact of it?

I'm collecting data on powerfully persuasive speech acts; it's part of a dangling thread of curiosity after GPT-2 (a new and fairly powerful text generation algorithm). I'm skeptical of the danger of mind-warping sentences as sometimes presented in fiction, or AI scenarios, and trying to get a sense of what the territory is like.

I've made a form to collect personal examples of things-someone-said that caused you to seriously change some belief or behavior. An easy example would be if someone declared that they love you, and this caused you to suddenly devote a lot more (or a lot less!) time and attention to them as a person.

If you have five minutes, my goal for this form is 1000+ responses and your own response(s) will help with that. All replies are anonymous, and there's a place for you to restrict how the information is used/state confidentiality desires. You can also fill it out more than once if you want.

https://goo.gl/forms/39x3vJqNomAome382 is the link to the form, if you want to share with anyone else; I'm happy to have this spread around wherever.

Edit: the title was misleading, i didn't ask about a rational agent, but about what comes out of certain inputs in Bayes theorem

Eliezer and others talked about how a Bayesian with a 100% prior cannot change their confidence level, whatever evidence they encounter. that's because it's like having infinite certainty. I am not sure if they meant it literary or not (is it really mathematically equal to infinity?), but assumed they do.

I asked myself, well, what if they get evidence that was somehow assigned 100%, wouldn't that be enough to get them to change their mind? In other words -

If P(H) = 100%

And P(E|H) = 0%

than what's P(H|E) equals to?

I thought, well, if both are infinities, what happens when you subtract infinities? the internet answered that it's indeterminate*, meaning (from what i understand), that it can be anything, and you have absolutely no way to know what exactly.

So i concluded that if i had understated everything correct, than such a situation would leave the Bayesian infinitely confused. in a state that he has no idea where he is from 0% to a 100%, and no amount of evidence in any direction can ground him anywhere.

Am i right? or have i missed something entirely?

*I also found out about Riemann's rearrangement theorem which, in a way, let's you arrange some infinite series in a way that equals whatever you want. Dem, that's cool!

This blog post is composed as following:

1. Review of Prisoners Dilemma
2. Explanation of Game of Chicken by comparing it to Prisoners Dilemma
3. Blackmail is a Game of Chicken
4. Why we should care about blackmail/Game of Chicken
5. What to do? Iterated Game of Chicken?
   

You are encouraged to skip ahead to the part that interests you

1. Review of Prisoners Dilemma

Prisoners Dilemma is a class of two player games which can represent for example mutual beneficial cooperation, or the tragedy of the commons. I don't think it is controversial to say that this class of games are important in almost any multi-agent scenario.

In a Prisoners Dilemma , each player gets to choose between two actions, usually called "cooperate" and "defect". Further more the payoffs haved to fulfill the following:

• Holding my action constant it is better for me if you cooperate.
• Holding your action constant, it is better for me if I defect.
• Cooperate-cooperate is Pareto optimal (even when including mixed strategies).

Example of a payout matrix for Prisoners Dilemma:

In this particular example, cooperate corresponds to spending one of your own utility to give the other player two utility, and defect corresponds to doing nothing. This can represent a situation with the possibility of mutual benefit from cooperation, but where it is possible to win even more (at the other players expense) by cheating.

But we can also consider a negative game:

Here cooperate is doing nothing, while defect corresponds to gaining one utility for yourself while costing the other player two utility. This can represent burning the commons (if the players defect) or not (if they cooperate).

2. Explanation of Game of Chicken by comparing it to Prisoners Dilemma

Just like Prisoners Dilemma, Game of Chicken is a two player game, where each player can choose between two actions. These actions are typically called "swerve" and "straight", but in this blog post I will instead call the two actions "cooperate" and "defect" as to more easily compare with Prisoners Dilemma.

Also the same as Prisoners Dilemma: In Game of Chicken, I get the best payout if I defect and you cooperate (and vice versa). The difference is that conditional on you defecting, it is better for me if i cooperate.

A two action, two player game is a Game of Chicken if:

• Holding my action constant it is better for me if you cooperate.
• If you cooperate it is better for me if I defect.
• If you defect is better for me to cooperate.
• Cooperate-cooperate is Pareto optimal (even when including mixed strategies).

Furthermore, defect-defect is traditionally super bad for both players. But I would not say that this is a necessary condition for something to be a Game of Chicken.

Example payoff matrix:

The interesting part here is that I can pressure you to cooperate by credibly convincing you that I will defect. In other words, there is a first mover advantage, the first one to precommit to defecting will win against a rational player. However, this fact is of course known by every rational agent, so it might be a rational move to pre-commit to always defect in such games, no mater what. Then again, if two players with such commitments meet, they will both lose.

3. Blackmail is a Game of Chicken

I think that this is easiest explained by just writing out an example payout matrix

If the the blackmailed player gives in, then they pay two utility to give the other player one utility. If the blackmailed player doesn't give in the blackmailer will carry out the threat which is costing both players ten utility. If the the blackmailer doesn't actually blackmails, than nothing happens.

Compare this to the example payout matrix of Game of Chicken. The blackmail payout matrix is not exactly the same, but I claim that in essence this is the same game. If you can handle Game of Chicken then you can handle blackmail both as the blackmailer and the blackmailed.

Not all blackmail is a Game of Chicken. If there is not cost in carrying out the threat then we are in a different type of situation. However I expect this to be rare. It seems unlikely to me that there is no opportunity cost at all in carrying out the threat. Further more, even if costless threats exists in some situations this does not invalidate the argument for considering those blackmail situation where there is a cost to the blackmailer to carryout the threat.

If the blackmailer gains utility by carrying out the threat then I would argue that it is not exactly blackmail anymore. If I have an action that I can take that would help me but hurt you and I ask you for some compensation for refraining from taking this action, then this is more like a value trade than a blackmail.

4. Why we should care about blackmail/Game of Chicken

Prisoners Dilemma receives a lot of attention because this class of games represents an important type of situation in most multiplayer environments. I claim that this is also true for Game of Chicken.

In any situation where one agent (A) has the ability to use up some of its own resources to impose a cost on another agent (B), then A can choose to blackmail B, thus creating a Game of Chicken like situation. And if A thinks that it can win this game, then it will be tempted to engage in blackmail.

If you expect that:

• It is important to build AI's that can act well in multi-agent situations (e.g. because there will be several simultaneous AIs that are similarly powerful, or there will be acausal trade and threats between agents in different universes simulating each other)

and

• Toy model such as Prisoners Dilemma are useful

then you should also care about Game of Chicken.

5. What to do? Iterated Game of Chicken?

What should we do about these insights? I am not sure yet. But one possible directions is to study iterated Game of Chicken.

Abram Demski argues that In Logical Time, All Games are Iterated Games. Basically if agents are simulating each other then this is sort of equivalent to the agent playing an iterated game.

Question for the comment section: What would be the winning strategy in iterated Game of Chicken?

I might run a tournament with different strategies.

This post was written with the support of the EA Hotel

I have a productivity scale I've used to log productivity data for several years. It's a subjective rating system from 1 to 10, and looks something like

1. can’t do anything, even reading. Worktime 0%.

2. can force myself to read or work, but I can barely parse anything. Worktime 5%.

...

5. I don’t want to work and am easily distracted, but I’m getting stuff done. 50%

6. Some mild distractions, but I can stick to a pomodoro timer. Worktime 60%.

...

10. Insane art-level hyperfocus. Worktime 100%.

At the end of each workday I would record how well I thought I'd done by this scale.

I'd been dissatisfied with this for a while – there was no way my brain was accurately tracking what percentage of time I was working, these descriptions are not well defined, don't cleanly map to some level of productivity. I can't prevent internal mapping drift, where my standards slowly rise or fall, such that a day I mark as productivity=4 this week is actually much more productive than a day I marked at 4 several years ago.

I'm invested in having good measurements, because I've been iterating on antidepressants and ADD meds for years, and having data on which ones are working on what metrics (I also track mood, exercise, sleep, social time) would be very useful for having a better life.

So I wrote a small Python script that tracked how much time I spent working at my job. Every time I took a break or went back to work, I'd mark it. If I noticed I'd started working during breaktime or zoning out during worktime, I'd 'revert' however many minutes I thought it had been to be of the other type. I also had 'dead' time where I wasn't getting anything done for reasons unrelated to my productivity, which I used to mark meetings or lunch breaks. At the end of the session, it would spit out a summary of how much time I'd spent working vs resting. This was a much more accurate, quantitative way of measuring what I wanted, or so I thought. I used it for a month and a week before I stopped.

Here's why I quit it.

• As is frequently the case, using part of a system to monitor that system changes that system enough that the output of the monitoring is less useful. On bad attention days, I would be switching back and forth between work every two minutes. The work/break context switches were costlier, and seeing how short my work periods were lowered my mood.
• The varying difficulty of tasks threw off the measurement. Some days I'd have monotonous but easy work, sometimes I'd have one tricky complicated thing that took a lot of brainpower and persistence. When I was using the subjective scale, I'd adjust my score for perceived task-easiness. But when I was using the time tracker, a day that was spent on "going through the codebase and adjusting every method call to do a new thing" would be logged as productive, even though I could have done that on that even on a bad day.
• My personal productivity scale ranged between 0% and 100% productivity, and my ratings usually fall between 2 (subjective 5% work time) to 6 (60%). But my time-tracked work time usually hovered around 50%. I have an internal sense of "I haven't done enough work, I really need to do something" that kicks in and makes me do work-like activities badly, slowly, and ineffectively. For example, I'll read some documentation, staring at one sentence at a time, forcing myself to process it before moving onto the next one. That counts as work by the tracker time -- I'm certainly not resting -- and I can fill half my workday with that. At the end, the tracker will say I worked 50% of the time, but my subjective scale would say it was a 2. And I think my subjective scale is more correct.

I considered the idea of rating work periods every time I ended one, so that after spending an hour laboriously shoving sentences against my eyeballs, I'd indicate to the program that "taking break now; also, that last work period was only 10% of a real work period". But that goes right back to the problem of subjectively rated work, plus adds to the already-painful overhead I described in point 1.

That was an interesting lesson to learn. In the future, when I'm trying to measure something, I'll try to ask myself

• How will integrating monitoring into my system change my system?
• Describe a day that the proposed monitoring system would give a high score, but actually should have a bad score. How common do you expect these days to be?
• Describe a day that would get a low score should have a high score, etc.
• How much overhead do you expect this to add?

Programmers generally distinguish between “imperative” languages in which you specify what to do (e.g. C) versus “declarative” languages in which you specify what you want, and let the computer figure out how to do it (e.g. SQL). Over time, we generally expect programming to become more declarative, as more of the details are left to the compiler/interpreter. Good examples include the transition to automated memory management and, more recently, high-level tools for concurrent/parallel programming.

It’s hard to say what programming languages will look like in twenty or fifty years, but it’s a pretty safe bet that they’ll be a lot more declarative.

I expect that applied mathematics will also become much more declarative, for largely the same reasons: as computers grow in power and software expands its reach, there will be less and less need for (most) humans to worry about the details of rote computation.

What does this look like? Well, let’s start with a few examples of “imperative” mathematics:

• Most grade-school arithmetic: it’s explicitly focused on computation, and even spells out the exact steps to follow (e.g. long division).
• Gaussian reduction, as typically taught in a first-semester linear algebra class. It’s the undergrads’ version of grade-school arithmetic.
• Most of the computation performed by hand in physics, engineering and upper-level econ courses & research, i.e. algebra/DEs/PDEs.

Contrast to the declarative counterparts:

• Figure out what arithmetic needs to be done (i.e. what numbers to plug in) and then use a calculator
• Set up a system of linear equations, then have python or wolfram invert the matrix
• Choose which phenomena to include in a model, set up the governing equations, then use either numerical simulation (for pretty graphs) or a computer algebra system (for asymptotics and scaling relations).

In the declarative case, most of the work is in formulating the problem, figuring out what questions to ask, and translating it all into a language which a computer can work with - numbers, or matrices, or systems of equations.

This is all pretty standard commentary at the level of mathematics education, but the real importance is in shaping the goals of applied mathematics. For the past century, the main objectives of mathematical research programs would be things like existence & uniqueness, or exhaustive classification of some objects, or algorithms for solving some problem (a.k.a. constructive solution/proof). With the shift toward declarative mathematics, there will be more focus on building declarative frameworks for solving various kinds of problems.

The best example I know of is convex analysis, in the style taught by Stephen Boyd (course, book). Boyd’s presentation is the user’s guide to convex optimization: it addresses what kinds of questions can be asked/answered, how to recognize relevant applications in the wild, how to formulate problems, what guarantees are offered in terms of solutions, and of course a firehose of examples from a wide variety of fields. In short, it includes exactly the pieces needed to use the tools of convex analysis as a declarative framework. By contrast, the internals of optimization algorithms are examined only briefly, with little depth and a focus on things which a user might need to tweak. Complicated proofs are generally omitted altogether, the relevant results simply stated as tools available for use.

This is what a mature declarative mathematical framework looks like: it provides a set of tools for practitioners to employ on practical problems. Users don’t need to know what’s going on under the hood; the algorithms and proofs generally “just work” without the user needing to worry about the details. The user’s job is to understand the language of the framework, the interface, and translate their own problems into that language. Once they’ve expressed what they want, the tools take over and handle the rest.

That’s the big goal of future mathematical disciplines: provide a practical framework which practitioners can use to solve real-world problems in the wild, without having to know all the little details and gotchas under the hood.

One last example, which is particularly relevant to me and to ML/AI research. One of the overarching goals of probability/statistics/ML is to be able to code up a generative model, pass it into a magical algorithm, and get back parameter estimates and uncertainties. The “language” of generative models is very intuitive and generally easy to work with, making it an excellent interface to a declarative mathematical toolkit. Unfortunately, the behind-the-scenes part of the toolkit remains relatively finicky and inefficient. As of today, the “magical algorithm” part is usually MCMC, which is great in terms of universality but often super-exponentially slow for multimodal problems, especially in high dimensions, and can converge very slowly even in simple unimodal problems. It’s not really reliable enough to use without thinking about what’s under the hood. Better mathematical tools and guarantees are needed before this particular framework fully matures.

If anyone has other examples of maturing or up-and-coming declarative mathematical frameworks, I’d be very interested to hear about them.

There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?

1. Insufficient time/resources for AI safety (for example caused by intelligence explosion or AI race)
2. Insufficient global coordination, leading to the above
3. Misspecified or incorrectly learned goals/values
4. Inner optimizers
5. ML differentially accelerating easy to measure goals
6. Paul's "influence-seeking behavior" (a combination of 3 and 4 above?)
7. AI generally accelerating intellectual progress in a wrong direction (e.g., accelerating unsafe/risky technologies more than knowledge/wisdom about how to safely use those technologies)
8. Metaethical error
9. Metaphilosophical error
10. Other kinds of philosophical errors in AI design (e.g., giving AI a wrong prior or decision theory)
11. Other design/coding errors (e.g., accidentally putting a minus sign in front of utility function, supposedly corrigible AI not actually being corrigible)
12. Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)
13. Human-controlled AIs ending up with wrong values due to insufficient "metaphilosophical paternalism"
14. Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity
15. Intentional corruption of human values
16. Unintentional corruption of human values
17. Mind crime (disvalue unintentionally incurred through morally relevant simulations in AIs' minds)
18. Premature value lock-in (i.e., freezing one's current conception of what's good into a utility function)
19. Extortion between AIs leading to vast disvalue
20. Distributional shifts causing apparently safe/aligned AIs to stop being safe/aligned
21. Value drift and other kinds of error as AIs self-modify, or AIs failing to solve value alignment for more advanced AIs
22. Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs
23. Gradual loss of influence due to insufficient competitiveness of humans & human-aligned AIs
24. Utility maximizers / goal-directed AIs having an economic and/or military competitive advantage due to relative ease of cooperation/coordination, defense against value corruption and other forms of manipulation and attack, leading to one or more of the above
25. In general, the most competitive type of AI being too hard to align or to safely use
26. Computational resources being too cheap, leading to one or more of the above

(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)

This is a linkpost for https://app.grasple.com/#/level/1669

Every Thursday for 4 weeks, we will be posting lessons about Iterated Distillation and Amplification. They're largely based on Paul Christiano's sequence here on LW. He graciously allowed us to use his work.

This is the third section, containing one lesson synthesizing the blog posts detailing the scheme and the first of our video series on the scheme, more of which will be coming in the next few weeks. The video can also be found on Youtube .

Note that access to the lessons requires creating an account here.

Have a nice day!

When it will appear? (My guess is 2020).

Will it be created by OpenAI and will it be advertised? (My guess is that it will not be publicly known until 2021, but other companies may create open versions before it.)

How much data will be used for its training and what type of data? (My guess is 400 GB of text plus illustrating pictures, but not audio and video.)

What it will be able to do? (My guess: translation, picture generation based on text, text generation based on pictures – with 70 per cent of human performance.)

How many parameters will be in the model? (My guess is 100 billion to trillion.)

How much compute will be used for training? (No idea.)

I've been thinking how I can improve my accuracy predicting events of personal interest (e.g., "Will my landlord get the washing machine fixed within the next two weeks", or "Will my parent die this year" for a more extreme example). Betting markets will not help me with that.

At first I thought about creating dedicated software that gathers such predictions, the final outcomes of predicted events, and presents their accuracy so that the user can spot bias. Then I realised a simple spreadsheet might suffice to gather data at first and assess how useful this is. And if the need arises in the future, it should be easy to import into dedicated software, provided that all the relevant data is already there.

Does anyone track their personal predictions? If so, what methodology do you use, and did it allow you to improve your accuracy?

As an RFC, here's the spreadsheet layout I have on mind:

• Tags: (value 0 or 1):
• Health
• Finance
• Interpersonal relations
• ...
• Date of the forecast
• Event (e.g., "My landlord will get the washing machine fixed within the next two weeks"). I'm planning to formulate them so that "yes" is always the desired outcome, so that it's easy to spot if I'm reliably too optimistic or pessimistic.
• Estimated probability
• Deadline of the forecast
• Outcome (value 0 or 1, filled after the deadline of the forecast, or when the answer is known sooner)