Douglas Reay graduated from Cambridge University (Trinity College) in 1994. He has since worked in the computing and educational sectors.
I wonder how much an agent could achieve by thinking along the following lines:
Big Brad is a human-shaped robot who works as a lumberjack. One day his employer sends him into town on his motorbike carrying two chainsaws, to get them sharpened. Brad notices an unusual number of the humans around him suddenly crossing streets to keep their distance from him.
Maybe they don't like the smell of chainsaw oil? So he asks one rather slow pedestrian "Why are people keeping their distance?" to which the pedestrian replies "Well, what if you attacked us?"
Now in the pedestrian's mind, that's a reasonable response. If Big Brad did attack someone walking next to them, without notice, Brad would be able to cut them in half. To humans who expect large bike-riding people carrying potential weapons to be disproportionately likely to be violent without notice, being attacked by Brad seems a reasonable fear, worthy of expending a little effort to alter walking routes to allow running away if Brad is violent.
But Brad knows that Brad would never do such a thing. Initially, it might seem like asking Brad "What if 2 + 2 equalled 3?"
But if Brad can think about the problem in terms of what information is available to the various actors in the scenario, he can reframe the pedestrian's question as: "What if an agent that, given the information I have so far, is indistinguishable from you, were to attack us?"
If Brad is aware that random pedestrians in the street don't know Brad personally, to the level of being confident about Brad's internal rules and values, and he can hypothesise the existence of an alternative being, Brad' that a pedestrian might consider would plausibly exist and would have different internal rules and values to those of Brad yet otherwise appear identical, then Brad has a way forwards to think through the problem.
On the more general question of whether it would be useful for Brad to have the ability to ask himself: "What if the universe were other than I think it is? What if magic works and I just don't know that yet? What if my self-knowledge isn't 100% reliable, because there are embedded commands in my own code that I'm currently being kept from being aware of by those same commands? Perhaps I should allocate a minute probability to the scenario that somewhere there exists a lightswitch that's magically connected to the sun and which, in defiance of known physics, can just turn it off and on?", with careful allocation of probabilities that might avoid divide-by-zero problems, but I don't think it is a panacea - there are additional approaches to counterfactual thinking that may be more productive in some circumstances.
Suppose we ran a tournament for agents running a mix of strategies. Let’s say agents started with 100 utilons each, and were randomly allocated to be members of 2 groups (with each group starting off containing 10 agents).
Each round, an agent can spend some of their utilons (0, 2 or 4) as a gift split equally between the other members of the group.
Between rounds, they can stay in their current two groups, or leave one and replace it with a randomly picked group.
Each round after the 10th, there is a 1 in 6 chance of the tournament finishing.
How would the option of neutral (gifting 2) in addition to the build (gifting 4) or break (gifting 0) alter the strategies in such a tournament?
Would it be more relevant in a variant in which groups could vote to kick out breakers (perhaps at a cost), or charge an admission (eg no share in the gifts of others for their first round) for new members?
What if groups could pay to advertise, or part of a person’s track record followed them from group to group? What if the benefits from a gift to a group (eg organising an event) were not divided by the number of members, but scaled better than that?
What is the least complex tournament design in which the addition of the neutral option would cause interestingly new dynamics to emerge?
If you have a dyson swarm around a star, you can temporarily alter how much of the star's light escape in a particular direction by tilting the solar sails on the desired part of the sphere.
If you have dyson swarms around a significant percentage of a galaxy's stars, you can do the same for a galaxy, by timing the directional pulses from the individual stars so they will arrive at the same time, when seen from the desired direction.
It then just becomes a matter of math, to calculate how often such a galaxy could send a distinctive signal in your direction:
Nm (number of messages)
The surface area of a sphere at 1 AU is about 200,000 times that of the area of the sun's disc as seen from afar.
Lm (bit length of message)
The Aricebo message was 1679 bits in length.
Db (duration per bit)
Let's say a solar sail could send a single bit every hour.
We could expect to see an aricebo length message from such a galaxy once every Db x Lm x Nm = 40 millennia.
Of course messages could be interleaved, and it might be possible to send out messages in multiple directions at once (as long as their penumbra don't overlap). If they sent out pulses at the points of a icosahedron and alternated sending bits from the longer message with just a regular pulse to attract attention, 200 years of observation should be enough to peak astronomer's interest.
But would such a race really be interested in attracting the attention of species who couldn't pay attention for at least a few millennia? It isn't as if they'd be in a rush to get an answer.
Alice and Bob's argument can have loops, if e.g. Alice believe X because of Y, which she believes because of X. We can unwind these loops by tagging answers explicitly with the "depth" of reasoning supporting that answer
A situation I've come across is that people often can't remember all the evidence they used to arrive at conclusion X. They remember that they spent hours researching the question, that they did their best to get balanced evidence and are happy that they conclusion they drew at the time was a fair reflection of the evidence they found, but they can't remember the details of the actual research, nor are the confidence that they could re-create the process in such a way as to rediscover the exact same sub-set of evidence their search found at that time.
This makes asking them to provide a complete list of Ys upon which their X depends problematic, and understandably they feel it is unfair to ask them to abandon X, without compensating them for the time to recreate an evidential basis equal in size to their initial research, or demand an equivalent effort from those opposing them.
(Note: I'm talking here about what they feel in that situation, not what is necessarily rational or fair for them to demand.)
If, instead of asking the question "How do we know what we know?", we ask instead "How reliable is knowledge that's derived according to a particular process?" then it might be something that could be objectively tested, despite there being an element of self-referentiality (or boot strapping) in the assumption that this sort of testing process is something that can lead to a net increase of what we reliably know.
However doing so depends upon us being able to define the knowledge derivation processes being examined precisely enough that evidence of how they fare in one situation is applicable to their use in other situations, and upon the concept of there being a fair way to obtain a random sample of all possible situations to which they might be applied, despite other constraints upon the example selection (such as having a body of prior knowledge against which the test result can be compared in order to rate the reliability of the particular knowledge derivation process being tested).
Despite that, if we are looking at two approaches to the question "how much should we infer from the difference between chimps and humans?", we could do worse than specify each approach in a well defined way that is also general enough to apply to some other situations, and then have a third party (that's ignorant of the specific approaches to be tested) come up with several test cases with known outcomes, that the two approaches could both be applied to, to see which of them comes up with the more accurate predictions for a majority of the test cases.
All 8 parts (that I have current plans to write) are now posted, so I'd be interested in your assessment now, after having read them all, of whether the approach outlined in this series is something that should at least be investigated, as a 'forgotten root' of the equation.
Two similar ideas:
There is a group evolutionary advantage for a society to support punishing those who defect from the social contract.
We get the worst democracy that we're willing to put up with. If you are not prepared to vote against 'your own side' when they bend the rules, that level of rule bending becomes the new norm. If you accept the excuse "the other side did it first", then the system becomes unstable because there are various baises (both cognitive, and deliberately induced by external spin) that make people more harshly evaluate the transgressions of other, than they evaluate those of their own side.
This is one reason why a thriving civil society (organisations, whether charities or newspapers, minimally under or influenced by the state) promotes stability - because they provide a yardstick to measure how vital it is to electorally punish a particular transgression that is external to the political process.
A game of soccer in which referee decisions are taken by a vote of the players turns into a mob.
shminux wrote a post about something similar:
possibly the two effects combine?
Other people have written some relevant blog posts about this, so I'll provide links: