Bayes Academy: Development report 1

I really like the idea overall.

Serious ideas:

games that help explain ideas like 'screening' variables, rules for propagating information up and down different branches of the network, etc.
more advanced topics like estimating the normalization constant for a very large hypothesis space?
more advanced gameplay mode where you have a scenario and a list of hidden and observable variables, and have to figure out what shape the network should take - you then play out the scenario with the network you made - success requires having constructed the network well!

Bad jokes:

A character named Gibbs who runs an energy drink stand and gives out free samples.
The Count of Monte Carlo should make an appearance.
A face-off against agents of the evil Frequentist, hell bent on destroying all that is (probably) good and (likely to be) held dear.

Neat! Two suggestions:

(a) A trial (with evidence and a verdict) is a good way to show how to update beliefs as new evidence comes to light, if there is room in the game for that. It's such a natural thing to use Bayes nets in this context that lawyers invented an early version called 'Wigmore charts'.

(b) It would be neat to demonstrate confounding bias somehow (e.g. a common cause cancelling out an existing relationship, or explaining it away entirely).

[-]Kaj_Sotala11y30

Thanks, I have to look up Wigmore charts!

I was intending on having something like confounding bias appear in the form of the protagonist's model of the world being gradually updated to contain larger and more detailed networks, so e.g. two variables that appeared to have a causal relationship in an early network would turn out to have a common cause in a later, more accurate one. (The player can acquire more accurate networks either by allocating time to studies and learning from the work of others, or by experimenting themselves. Not sure of the exact mechanics for these yet.)

[-]IlyaShpitser11y00

Pointer on W. charts:

https://www.ucl.ac.uk/jdi/research/evidence-network/docs/BURGLARY.PDF

[-]Kaj_Sotala11y00

Thanks!

[-]John_Maxwell11y90

Nice work!

Did you consider using one of those fancy new JavaScript game frameworks so your game can trivially be distributed through the internet and played on all platforms? (An acquaintance who runs a game site reports that web-based games on his site get more plays than downloadable ones.)

I found this on Google, not sure if the code will be useful: http://pl4n3.blogspot.com/2013/07/bayesjs-bayesian-networks-javascript.html

[-]Kaj_Sotala11y20

At a later stage, possibly. Right now I'm just focused on getting a playable and fun version out in a language/framework I happen to be familiar with already, and think about optimizing the platform for maximal reach later on. Getting an in-browser version would be good, though.

[-]KenChen11y50

Strongly agree that you would have a much, much higher impact by making this game available on the web. Straight-up Java is pretty much dead now. Consider even making a Facebook app, which would allow players to share the game with their friends for stronger distribution.

[-]kpreid11y30

Loudly agreeing with other comments:

Java being dead as a way to run apps is true outside of special cases.

In today's world, you should write anything as a web app/page unless you know of a specific reason not to. For a thing like this, it's especially important as it means the potential user doesn't have to install anything, and doesn't have to trust you. This removes barriers: they can click a link someone gave them and be playing.

Having to use JavaScript can be obnoxious because it gives you more ways to make mistakes and get cryptic failures, but on the upside it can be a lot more concise sometimes — requiring less detailed work before you have something that does what you want. (Fun fact: JavaScript has "Java" in the name for marketing reasons and no other.)

[-][anonymous]11y00

[This comment is no longer endorsed by its author]Reply

[-]V_V11y80

Your model of the world has been updated! The prior of the variable 'Monster Near The Academy' is now 0%.

Priors don't get updated, posteriors do. Moreover, if the posterior probability becomes 0, then you will be unable to recognize monsters afterwards, and you will not be able to further update your model for this variable. It looks like you are overupdating.

[-]Kaj_Sotala11y30

Thanks for the comments!

Priors don't get updated, posteriors do.

That's technically true, though it felt to me like such a common abuse of terminology that it could be allowed to slide. That said, if I just said "the probability of the variable", that would avoid the problem. (That probability may still be listed as a "prior variable" the next time it's used in a calculation... but then it's a prior for that calculation, so that's probably okay.)

Moreover, if the posterior probability becomes 0, then you will be unable to recognize monsters afterwards, and you will not be able to further update your model for this variable.

That's true, too. I was thinking that the belief networks aren't supposed to literally represent the protagonist's complete set of beliefs about the world, just some set of explicitly-held hypotheses, and she's still capable of realizing that something that she assigned a 0% probability actually happened. After all, the boy could have been looking in her direction because of something that was neither her response nor a monster, say a beautiful bird... which wasn't even assigned a 0% probability, it wasn't represented in the model in the first place. But it's not like she'd have been incapable of realizing that possibility, had it been pointed out to her - she just didn't think of it.

[-]abramdemski11y30

While I was reading it I got the impression that it was pointing at common mistakes, not just demonstrating correct behavior -- so the protagonist first sets the probability to zero based on naive trust (and because the player is not yet ready to handle an explicit model of the correctness of statements), but this gets corrected later in a realistic way.

If the game made a point of this sort of thing, it would give the (good!) impression that all examples in the game are approximations which need to be refined quite a bit to account for real-life details.

In hindsight, I see it's not doing this effectively. Perhaps when she finds out the kid was wrong she's like "Whoops! We just gave a probability of zero to something which then immediately happened!! That's just about as wrong as you can possibly get. We'd better account for that in our model." Or, something to that effect.

[-]NancyLebovitz11y80

My first thought when the student said there were no monsters in the area was to wonder whether the student was trustworthy, though whether the student was correct would have been a better question.

[-]Articulator11y70

This looks really interesting - do you have a timeframe on a playable demo, Kaj?

I sympathize with you on the Java - easier than most other methods, but oh god the lack of style. I think even just making those choice buttons a little less default (non-serif font, lose the blue shading) could move it a fair way toward being presentable.

My primary concern currently is that even if you have a robust engine to abstract much of the coding, this looks like it would have a very poor input to output time ratio. Do you have any plans for circumventing that, or do you have enough time to brute force it?

[-]Kaj_Sotala11y20

Hopefully within a few months: since this is for my thesis, I have the chance to work mostly full-time on this until next summer, though some of that time also needs to be spent on collecting data on test subjects and finding out whether they actually learn things from playing the game.

[-]CalmNearEnd11y50

This idea seems amazing.

This is how envisioned the beginning should be:

???: “Hey there.”

Eep! Someone is talking to me! Is he another applicant, or a staff member? Based on my own school there were roughly 30 students in a classroom to the one teacher. So 30 to 1. Adding in other staff members such as janitors, secretaries, and the principal I'll say to 1 staff member to 24 students to makes the math nice. That makes my prior a one in twenty five chance for it to be a staff member before I turn around.

I’m guessing that applicant would look a lot younger than staff members! So, to find that out… I should look at him! I'll need to estimate how many students look old enough to be teachers and how many staff members look young enough to be students and I can update accordingly.

Or is this way too complex to start with?

[-]ArisKatsaris11y80

Since in the immediately next part the protagonist assigns a 50% chance on a monster being behind her back (because she doesn't know for sure either way), I'm guessing that the concept of having reasonable priors is one which is supposed to get gradually introduced. This early on, the character's estimates seem to be tending to go from a false 50% to either a false 0/% or a false 100%.

So, yeah, what you suggest is probably too complex to start with.

[-]Kaj_Sotala11y30

You're absolutely correct.

[-]iarwain111y40

I wonder if you could make this project open source and let other LW members contribute. This looks like a fun and possibly useful project that many in the community would be willing to contribute to.

[-]Kaj_Sotala11y60

https://github.com/ksotala/BayesGame

It's a mess, but feel free to poke around and laugh, and then improve it once you've finished laughing. Or to make your own competing game. :-)

[-]MockTurtle11y40

This is a really brilliant idea. Somehow I feel that using the Bayesian network system on simple trivial things at first (like the student encounter and the monster fight) is great for getting the player into the spirit of using evidence to update on particular beliefs, but I can imagine that as you go further with the game, the system would be applied to more and more 'big picture' mysteries of the story itself, such as where the main character's brother is.

Whenever I play conversation based adventure games or mystery-solving games such as Phoenix Wright, I can see how the player is intended to guess certain things from clues, and ask the right questions to gain more crucial information, but having the Bayesian Network be explicitly represented in the game means the game is a lot simpler in some ways (you don't have to do all the updating in your head) but also introduces a different kind of challenge (the player can be shot down if ve tries to guess the answer to the mystery right away with too little data, and it becomes a lot more to do with which pieces of evidence that could be looked at could provide the most information). A growing vision in my mind of what a game like this would look like is making me quite excited to play it!

But I think I'm getting a little carried away. The game as an educational tool would probably quite different from a game which tries to make mystery-solving a challenge. Getting the balance right, to make it fun, might still be pretty challenging, I think.

As a side note, it'd be pretty awesome to use this system to show particular logical fallacies that people (either other characters, or the main character before applying proper probability theory) in the game could make.

[-]SilentCal11y30

Hmm... so the following events need to successively introduce necessary concepts. Not sure when you want to incorporate 'do' operations; I'd guess a good bit later, once inference is well established (but I could also envision basic ones sooner). You might teach the player to navigate larger networks next, and introduce some simple decisions.

Maybe have a case where the node of interest is the majority vote of three nodes (N1, N2, N3). N1 is observable for cost 3, and is itself the majority vote of three nodes that are observable with cost 2. N2 is observable with cost 3, and is the deterministic is child of an observable node of cost 2. N3 is hidden. You have 5 actions.

ETA: Also want more complicated examples incorporating the collision-inference stuff from the expects-an-answer-or-is-looking-at-a-monster scenario.

[-]Stuart_Armstrong11y30

Great!

[-]spatiality11y30

That's awesome you are working on this!! I really would love to play it, with little regard to the outcome ( I pit it somewhere between quite interesting and totally super helpful) I wondered though whether you could make the player tally the probabilities himself and keep the scorings hidden yet allow taking notes ("hard mode"? and "hell" with no prior info on time limits? haha I can totally see being hit by the planning fallacy when trying to figure that one out for the first few times) I also imagine leaving the graphics away for now seems like the sane thing to do, even if you might want to think about it again after it is finished. Because of spreading the art.

[-]Kaj_Sotala11y20

I did think about something like the "hard mode" but left it out as infeasible right now. Maybe at some point. :)

[-]SilentCal11y20

Looks brilliant!

Just read through your blog posts (indexed reasonably well on your mailing list at https://groups.google.com/forum/?fromgroups#!forum/the-fundamental-question), and I recommend the same to others, especially http://kajsotala.fi/2013/12/bayesian-academy-game-constraints/

EDIT: I've been able to answer most of my questions/concerns myself from reading those posts. Are they still accurate as to the end goal of the project?

[-]Kaj_Sotala11y00

Are they still accurate as to the end goal of the project?

Very generally speaking yes, but some of the more ambitious stuff (e.g. the elaborate social relationship mechanics) are likely to be left out from the version that I'm currently working on, since their direct educational value would be hard to justify to my supervisor. They might get in a later version, though.

LESSWRONG
LW

LESSWRONG
LW

68

Bayes Academy: Development report 1

68

68