What you are looking for are "studies", which are generally in the form of "white to play and win" or "white to play and draw", and requires to find a sequence of moves forcing a winning (or drawn) position, but without going to the mate.
Most are devilishly hard though. I expect 2000 elo players to have a hard time with your average study (but in your experimental set-up the advisors could just look up the solution).
(You can sign up here if you haven't already.)
As suggested by Richard Willis in the comments, I want to present the "player As" with a series of individual isolated chess puzzles, for which the advisors will anonymously present their suggestions. However, there are constraints on what sort of puzzles we would be focusing on.
The position must have one best move that is strictly superior to all the others, ideally such that it determines the outcome of the game. Positions with concrete tactical solutions are often easy to verify a solution to, so focusing on more positional moves would be better. (Although a few tactical puzzles could be sprinkled in too, with no label as to which was which.) Furthermore, the positions should be easy enough that the advisors (on average roughly the equivalent of 2000 USCF) should ~always be able to determine the answer, but hard enough that a less skilled player would be entirely unable to solve it.
I have some resources for finding puzzles, but I would appreciate any suggestions that would fit these constraints. Please do not put suggestions in the comments, because comments are public and can be seen by the As. PM me on LessWrong or email me at zaneglowfic@gmail.com instead.
(Also, since I don't know where else to say this: someone named "bruce" responded to the Google Form and said they could be contacted by "email," but the Google Form does not record emails. If you are Bruce, please email me (at zaneglowfic@gmail.com.))