This is a followup to the D&D.Sci post I made nine days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted).

Ruleset

Skills and Competence

The success of a hero is decided by six hidden variables: Int, Wis, Cha, Str, Con, and Dex^[1]. These are used to produce Competence via the formula

Competence = 8*Int + 9*Wis + 10*Cha + 10*min(Int,Wis,Cha) + 11*max(Str,Con,Dex) + 11*Str+12*Con + 13*Dex[2]

Competence, Luck, and Success

The Challenge of saving a world is given by

Challenge = 1d20 + 1d40 + 1d60 + 1d80 + 1d100 + 1d200 + 1d400

If a hero’s Competence exceeds the Challenge, they succeed.

27.1% of generated heroes are Lucky. A Lucky hero, if they don’t succeed fairly, rolls a d8; on an 8, they succeed anyway through sheer force of Fate. This makes Luck a major factor for low-Competence heroes, but much less important to high-Competence ones.

The Fae and their Predictions

Amy

Amy believes fervently in Luck - particularly its ability to ameliorate otherwise hopeless situations - and nothing else. She predicts a 1% success rate for everyone who doesn’t have it, and a 99% success rate for everyone who does.

Bella and Liboulen

Bella (named for Peter Pan’s Tinkerbell) and Liboulen (named for Cleveland Quixotic’s Olliebollen) prioritize a hero’s mind over their body.

Bella is excessively cynical, and doesn’t care for Wisdom. Her predictions are given by

BellaPred = Cha+Int-1

Some bad experiences with over-clever heroes have left Liboulen falsely perceiving Intelligence as an active liability. Their predictions are given by

LiboulenPred = 40.7+5*Cha+Wis-Int

Linestra and Colleen

Linestra also prioritizes the mind over the body, linearly combining Int, Wis and Cha with a random element as below:

LinestraPred = 22.5 + 1.2*Int + 2.4*Wis + 3.6*Cha + 0.1*1d12 - 0.1

Colleen copies Linestra’s predictions, creating a nearly collinear column. Her guesses are Linestra’s, plus 1.7%; when Linestra predicts over 90% success chance, she adds a further 0.2%.

Fizz, Ziqual, and Ister

Fizz, Ziqual and Ister are physicalists, prioritizing physical stats over mental. Their predictions are given by

FizzPred = 41+2*Dex+min(Str,Con)
ZiqualPred = Str*Con - ((Str*Con)>12)*random.choice([0,1])
IsterPred = 50+Str

Holly

Holly takes a holistic approach. Her prediction makes use of all non-Luck stats, but also contains a random element strong enough to obfuscate the signal: this makes her the best single predictor (after adjusting for her systematic underestimation of strong heroes and underestimation of weak ones), but also makes her redundant with the other fae, whose predictions are collectively sufficient to perfectly reconstruct success chance without her.

HollyPred = 20+Str+Con+Dex+Int+Wis+Cha+1d12

Sick days

Whenever Linestra took a sick day, Colleen would predict 50% and hope no-one noticed. After the 5540^th batch, the fae got a live-in healer, and none of them have missed a day since then.

Choice of Chosen

The fae’s ranking of heroes is decided by taking a weighted average of their predictions of success chance, in which Holly’s predictions are considered twice as important as everyone else’s. This ranking is used to decide which heroes to send in the frequent circumstances when they have more available heroes than they do imperiled worlds. The result is a small but non-negligible selection bias, which particularly affects Amy: though Luck is important, a disproportionate number of Lucky-but-otherwise-underwhelming heroes are Chosen just because Amy’s exuberance outweighs everyone else’s sober assessments, producing the net result that Luck ends up looking almost useless in the set of heroes sent.

Answerkey

The best potential Chosen to choose is Candidate #11, with a success chance of ~90.2%; the next best is Candidate #19 (whom the fae would have chosen in your absence) with a success chance of ~84.2%, closely followed by Candidate #2 with a success chance of ~83.7%.

Holly’s predictions are the most useful when considered in isolation. The redundant fae are Colleen (copying Linestra’s work) and Holly (all information conveyed by her is conveyed with higher fidelity by someone else).

(Congratulations to everyone who figured any of this out, and particular congratulations to aphyer for figuring all of it out plus more of the generation structure than I thought I'd exposed.)

Reflections

This was planned as an ML challenge; the intended path to failure - aside from the perennial problem of picking poor hyperparameters - was judging characters’ usefulness using GBTs’ Feature Importance (will usually say Holly is the most valuable just because early-planted decision trees take the easiest route away from the starting value) and/or unpenalized linear regression’s coefficients (will invariably imply Colleen is more useful than everyone else combined due to collinearity effects’ pathological behavior); the intended path to success was identifying Holly by ablating advisors and noting changes in performance metrics, and identifying Colleen by Just Looking At The Data.

These paths were mostly left untravelled; shortcuts untaken, traps unsprung^[3]; even players who eventually resorted to ML seemed to lean strongly towards pure Analysis. I didn’t try to make this scenario a puzzle, but players played it as one regardless, and got further with human cleverness than they could have with the machines I made it for. I hope it was a fun one: feedback on this point, and on any other point, is greatly appreciated.

Scheduling

I plan to run my next scenario from the 20^th to the 30^th of next month . . . unless someone strongly prefers a different timing, and/or has one of their own they want to run in June.

I'll be delaying the next one a week because a frequent player expects to be busy over that timespan, and then delaying another week for my own convenience. Expect the next scenario from me to run starting on/around July 4th.

Heatwaves and other obligations have produced severe delays; apologies; next one's out when it's out.

1. ^{^}

   The generation process for these variables is mostly irrelevant to the task; feel free to look through the generation code if you want the full story.

2. ^{^}

   Everyone knows Dexterity is the most important trait a hero could possibly have.

3. ^{^}

   The one trick anyone managed to fall for was misidentifying Amy as redundant due to selection biases, which was an emergent effect I didn’t explicitly plan.