We're collecting human reasoning data for a study at Radboud University comparing how humans and large language models reason through social deduction scenarios.
We have previously published research on the limitations of LLMs at various reasoning tasks, and how applying cognitive hierarchy theory can improve them. Current research is trying to further understand LLMs' lack of strategic reasoning and their ability — or lack thereof — to work with missing and partial information.
The task uses Secret Mafia: a game that requires updating on partial information, modeling other agents' beliefs and incentives, and deciding under uncertainty. We're interested in the reasoning behind decisions: what beliefs participants form, how they weight evidence, and how they explain their inference process.
Each participant sees four game-state snapshots and is asked to: - Estimate the likely identities of players given the game state - Describe the action they would take given the current state - Explain their reasoning for the above
No prior knowledge of the game is needed. The rules are provided in full and all scenarios are self-contained.
This data forms the human baseline for a comparative analysis against LLM outputs, with the goal of characterising where human and AI reasoning diverge in tasks involving uncertainty, deception, and recursive social inference.
We think lesswrong is well-suited for this study — because participants who can articulate their reasoning explicitly produce substantially richer data, and we expect a higher proportion of such from lesswrong.
~15 minutes | Anonymous | Ethics approved| Radboud University / Donders Centre for Cognition
We're collecting human reasoning data for a study at Radboud University comparing how humans and large language models reason through social deduction scenarios.
We have previously published research on the limitations of LLMs at various reasoning tasks, and how applying cognitive hierarchy theory can improve them. Current research is trying to further understand LLMs' lack of strategic reasoning and their ability — or lack thereof — to work with missing and partial information.
The task uses Secret Mafia: a game that requires updating on partial information, modeling other agents' beliefs and incentives, and deciding under uncertainty. We're interested in the reasoning behind decisions: what beliefs participants form, how they weight evidence, and how they explain their inference process.
Each participant sees four game-state snapshots and is asked to:
- Estimate the likely identities of players given the game state
- Describe the action they would take given the current state
- Explain their reasoning for the above
No prior knowledge of the game is needed. The rules are provided in full and all scenarios are self-contained.
This data forms the human baseline for a comparative analysis against LLM outputs, with the goal of characterising where human and AI reasoning diverge in tasks involving uncertainty, deception, and recursive social inference.
We think lesswrong is well-suited for this study — because participants who can articulate their reasoning explicitly produce substantially richer data, and we expect a higher proportion of such from lesswrong.
~15 minutes | Anonymous | Ethics approved| Radboud University / Donders Centre for Cognition
https://questions.socsci.ru.nl/index.php/241752?lang=en
We intend to share the findings here once the paper is submitted. Happy to answer questions about methodology in the comments