Chess and cheap ways to check day to day variance in cognition

by KPier5 min read7th Jul 202128 comments

49

Rationality
Frontpage

Lately I've been playing chess. I'm not very good at it. I started because our four-year-old got into it, and I preferred not to lose to her before she had earned it. I play online on chess.com, which has instantaneous matchups with strangers at approximately your level; I mostly play 5-minute timed games, where each player has a 5-minute clock for all of their moves and loses if they run out of time. 

Doing this every day for a couple of months made some interesting patterns obvious. Firstly, I have good days and bad days, rather than just winning or losing mostly at random at my competence level. 

Bad days are predicted by some of the things you'd expect, like poor sleep, high stress, or emotional distraction, but sometimes I have a bad-day-as-measured-by-chess when I wouldn't have predicted it, or (more rarely) a strikingly good day when I thought I was likely sleep deprived.

Secondly, playing chess on a good day versus on a bad day allows a lot of access to the feeling of being slightly smarter or slightly stupider than usual. In general, I think my ability to do high-level cognition varies a lot day to day and even within a day, but it's often really hard to tell if I'm blocked on a problem because I'm a little low on ability to think about it; usually, a problem will be different from other problems in a bunch of ways, and it's hard to tell how much of the difficulty is that I'm fatigued/distracted/whatever. 

With chess, you're basically solving the same category of problem every time, with instantaneous feedback about whether your solution was any good, so I get unusually good felt-sense of where I would normally have seen something, but this time missed it, or alternatively where options and counteroptions are coming to me unusually quickly and have unusual depth and clarity. 

Thirdly, having a good day at chess seems pretty strongly associated with having a good day at high-cognition-requiring tasks in general. In fact, it seems like a better predictor of this than asking myself 'do I expect to be good at cognition today?'. 

On days when I do well at chess, I'm also likely to succeed at reading a complicated paper I previously bounced off because it was too hard, or at writing an article that has been blocked on loading all of it into working memory, or at having productive new thoughts if I dwell on an old problem for thirty minutes. I am not more likely to succeed at doing my taxes or other tasks that are motivation- rather than cognition- bottlenecked.

This could, obviously, be a self-fulfilling prophecy, or a placebo effect, but at least for me it makes playing chess a moderately valuable thing to do; in about ten minutes I can get a readout on how well my brain is working, and figure out from that which of my priorities for the week make sense to work on.

I am wildly uncertain how much this would generalize to other people, but it seems like a cheap experiment. Some suggestions, if you want to try it:
 

  • I do 5-minute games now, but for the first two months I did 10-minute games, and found the 5-minute ones pretty bad for me: I wasn't good enough at thinking about chess to think at all in the shorter time setting, and was just reflexively making moves on instinct, which is not very useful or interesting. Over time, I compressed/cached some stuff and now I get about the same experience from 5-minute chess I used to get from 10-minute chess. I recommend starting slower.
  • There's tons of advice about how to play chess - which openings to use, which countermoves to memorize for common openings by your opponent, etc. Don't learn any of it. Those things increase your ranking at chess, but the thing that's interesting here is chess as an environment in which you can see your brain function, and those things don't particularly help with that and are a time-sink of fairly arbitrary size. 

    You'll learn over time by losing and going 'okay, I'm not going to let that trick me again'. Your rating will probably eventually flatten out lower than the ratings of people who spend the time studying the meta, but that's not what you're using this for. Learn enough stuff about how the game is played that you have an idea of what criteria you should be using to evaluate potential moves - that is, "how do I know if this move leaves me in a stronger or weaker position" and then try to resist the urge to memorize openings or anything like that. 
  • chess.com offers more feedback than just "did you win the game or not" - in particular, there's an option to go through the game move by move and see what the smartest move would have been, and to revisit decisions you made that were particularly bad and try to make better ones. I do this sometimes. 

    They also rate your game by how closely your play adhered to 'perfect' play (what the chess engine would've done), on a 0 - 100 scale; unfortunately you can't take this number too literally as a 'how clever am I today' number because things like 'was there a long endgame where both sides had a bunch of forced/obvious moves' affect the scale a lot. I still use it for data, though.
  • If you try it, or if you already do this, please comment with: did you notice day to day or time-of-day variance in how well your brain seems to be working at chess? did it track with variance you'd noticed in other contexts? did it predict anything else? 

     

49

28 comments, sorted by Highlighting new comments since Today at 11:46 PM
New Comment

On a related note, one of the neater papers I've seen recently:

"Air Pollution and Adult Cognition: Evidence from Brain Training", La Nauze & Severnini 2021: using 4.6m scores from Lumosity brain-game players to measure PM2.5 cognitive effects: large but heterogeneous by task/age/experience---this may explain the inconsistencies of all the previous air-pollution/cognition studies.

I think once on LW I found checking the reaction time with some app as a proposed fast and uncomplicated benchmark to investigate cognition correlations with time of the day, sleep deprivation etc.

http://www.quantified-mind.com/ contains many cognition tests and games which can be done quickly.

I used https://humanbenchmark.com/ for quite a long time.

In addition to variance of opponents' skill, there is also a substantial luck factor. Any search algorithm that can't search the entire tree has uncertainty about how good moves are, which effectively translates into luck.

I think there's a good chance your observations are still valid (also because you have intuition about how well you play on top of results), but it is a possible factor.

You can get up to around 1600 ELO pretty much just by not making stupid one move blunders. I am ranked around 1550 and obvious blunders that I should have caught are consistently the reason I lose games.

I don't think this is in conflict with what I said. Blundering is itself a matter of luck. You can have both players play sufficiently reckless that they risk 1-move blunders, but then one person gets lucky that they've never made one. I assume you're familiar with the situation where you make a move that's only not a blunder due to where is something you didn't think about while making the move.

Also, I don't know how literal you meant your post, but I don't think it's true that you can get there by 'just' avoiding one-move blunders. If your positional play is sufficiently worse than your opponent's, you should lose even if your opponent blunders a minor piece at some point and you don't. I think it's more like, most people around 1500 are somewhat even in positional skill, and you can separate yourself from them by avoiding blunders.

(Out of curiosity (or perhaps because I'll challenge you), what time format are you 1550 in?)

I’m 1550 in blitz. Mostly I play on 5+3 on lichess. Pm me if you want my handle on that site.

I have a pretty big n dataset for Anki flashcards (and associated performance) and chess games performance that I could try measuring whether there's a predictive effect for long-term memory.

Here is what I did (n=524):

  • calculated performance rating for bullet games for each day
  • calculated anki accuracy (as measured by (1 - again%)) for each day
  • adjusted performance rating according to time by fitting an OLS model to predict perf rating with days since beginning of the dataset, then subtracting the model's prediction (should yield a normal distribution - this model had an R^2 of 0.4)
  • fit an OLS model to predict anki accuracy given the adjusted performance rating

This has an R^2 of 0.016, and the coefficient is ~5.5e-05 (though it is pretty significant). So a performance rating of 1000 higher than predicted only yields a boost of ~5% additional accuracy on anki. Since the adjusted performance rating has a standard deviation of 208 points, that means if you're having a "top cognition" day that's 2 std's above average, that's only 2% higher anki accuracy. Not a lot.

Of note: using a "locally smoothed" performance rating (where I smoothed the perf rating, then subtracted that from the perf rating to get a residual) yielded no significant correlation between anki accuracy and perf rating. Arguably this is a stronger bit of evidence - the above (naïvely) assumes that the perf rating goes up linearly with time, but this version is able to deal with plateaus and different slopes in increasing/decreasing rating.

I'm open for code/analysis review if anyone wants to double check my work.

I have the same experience with my level of play on chess.com - there are bad days when it feels like my hand is playing and some good days when I feel more like my head is playing. Not sure what external variable it is related to. In any case, you can play me if you befriend me there first ("Losna1", Losna - one, not Losna - capital i). I am 1400 on 10" time control, not much fun if you are more than 200 points far from that...

I also have crazy variation in my online chess performance. My rating varies by more than 400 points on some sites. 

Maybe worth pointing out that small differences in speed have a cumulative effect over the course of a game. But beyond speed I just seem to be blind to many possibilities on the off days. 

I vaguely planned to use online chess ratings to do a gwern style study on the effects of ginseng one day. 

The Elo on sites depends on the player base lots and lots. 

I swing between 1100 and 900 in chess ratings really obviously. Some days I'll win and win and win until I'm around 1100; other days I'll fall and fall until I dip into the 800s. A 200 ELO point difference is equivalent to winning the game 76% of the time, so this means that "best day me" would beat "worst day me" in 76% of head-to-head chess matches. 

I too play chess often and have had similar thoughts. But I wonder whether day-to-day variation in winning is mostly random, and any perceived correlation with other intellectual tasks is erroneous or self-fulfilling. You could do an experiment for a few weeks where you rate your ability on both each day - sometimes playing chess after, sometimes before other tasks - and measure the correlation. Of course it can’t be blinded so it may be self-fulfilling.

From another experiment I did on ways to improve sleep, I’ve found that self-perceived correlation between various ways to improve sleep (eg eating a banana before bed) and sleeping well turned out to be entirely illusory.

Also cf many years ago research showed that ‘form’ in sports (individual sportspeople going through extended periods of good or bad play) was almost entirely illusory and just random variation. Concurred by work I’ve been involved with on poker software - two identical computer programs playing each other can look like one is much better than the other for very long runs of hands.

How do you distinguish between your having a good day, and your opponent having a bad day?

One option is to do tactics puzzles instead of playing actual games. I don't know which is likely to correlate better with other aspects of mental performance.

This is easier to do by playing twenty 1-minute games.

Seems to test something different from 15 minute games.

I also play a few games of chess.com every morning and I have found it has the same use you describe.

I have noticed certain variations in my play depending on my emotional state. I am a casual player (1000 ELO on Lichess) but try to get in at least one game per day. It was a big deal for me to hit the 1000 mark, so I am very conscious of how much above it / below it I am at any time. When I am anxious or thinking about a past fight with my SO, or am stressed out about a current issue, I start losing games like crazy (And my ELO drops significantly). Once everything is fine and dandy, it starts climbing up again. I think chess is very much a game of extreme concentration and any intruding thoughts (which depending on the problem will penetrate with more / less ease) seep through and break that focus. 

The AI on lichess sometimes makes clearly worse moves than the ones I made. There is also room for much more in depth analysis like drop, fork, defensiveness, etc. I switched from chess to Anki and Amphetype because, while not nearly as fun, they also taught me a skill. I will get back to chess when I find an affordable automatic board.  Cognitive tracking is often discussed on reddit or the QS forums.

 Exist a few papers on the subject of chess as a test of cognition. 

"Using  within-player  comparisons,  we  find  a  statistically  and  economically  significant  decrease  in  performance  when  competing  online  compared  to  competing  offline."  Steffen Künn 2020

DB RCT Coffee causes "more reflective decision making processes. When not under time pressure," Franke 2017

I play a lot of chess and definitely notice a lot of day-to-day variation in how well I play. For me, the main factor seems to be if I had to wake up early or not. On days where I woke up naturally, I can pretty easily calculate out five or so moves, but on days where I had to wake up before the sun came up, it's much harder, and I can only clearly see two or three moves ahead. Caffeine seems not to affect this much.

I just want to say I came to the same conclusion as you (plaing 3-min games at chess.com several times a day, being in the 1250-1300 range) - quite a striking difference between good days and bad days, and only partly related to what I would expect (because of sleep deprivation etc.)
Also clearly better results in the evening than in the morning.
I have been thinking many times about statistically processing the results, but don't really have the time :)

It seems like it would be really interesting to study this quantitatively, recording data collected over time. If you (or anyone else) would be interested, I know a bunch of psychological research methods that could help.

How many games you play daily, and what time do you play them? If you have data from different times of the day, do you noticed any patterns? I once had a feeling that with some tasks at work I can "unblock" and really get the job done only after 5 or 6PM. But never put any serious effort into observing it for a longer period of time.

I've also observed something similar, at the decent-but-not-great club player level.

[+][comment deleted]2mo 1