A combination of the ideas in "binary search through spacetime" and "also look at your data":

If you know a previous time when the code worked, rather than starting your binary search at the halfway point between then and now, it is sometimes useful to begin by going ALL the way back to when it previously worked, and verifying that it does, in fact, work at that point.

This tests a couple of things:

1. Are you correct about when the code previously worked?
2. Did your attempt to recreate those conditions successfully recreate ALL of the relevant conditions?
   1. Does the bug depend on some external file or resource that you overlooked and haven't rewound back to the correct time?
   2. Does the bug depend on some detail of your test process that you didn't realize was relevant, and the reason it worked before was actually because you were testing it differently?
   3. Does your process for restoring your project to an earlier point even work?

If the bug still happens after you've restored to the "known working point", then you'll want to figure out why that is before continuing your binary search.

I don't always do this step. It depends how confident I am about when it worked, how confident I am in my restore process, and how mysterious the bug seems. Sometimes I skip this step initially, but then go back and do it if diagnosing the bug proves harder than expected.

I don't actually know if I count as a "senior developer", but I'm pretty convinced I count as a "senior debugger" given the amount of time I've spent chasing problems in other people's often-unfamiliar code.

These questions feel really hard to answer, mayber too general? When I try to answer them, I keep flying off into disconnected lists of heuristics and "things to look for".

Also, you don't give any examples of problems you've found hard, and I feel I may be answering at too simplistic a level. But...

How do you learn to replicate bugs, when they happen inconsistently in no discernable pattern?

The thing is that there is a discernable pattern. Once you've fixed the bug, you'll probably be able to say exactly what triggers it.

If you can't (yet) reproduce the bug, think about states of the program that could have led to whatever you're seeing.

You know what routine detected a problem^[1], and you usually know what routine produced bad output (it's the one that should have produced good output at that point).

Your suspect routine usually uses a relatively small set of inputs to do whatever it does. It can only react to data that it at least examines. So what does it use to do what it does? Where do its inputs come from, not necessarily in the sense of what calling routine passes them in, but in the sense of how they enter the overall program? What values can they take? What values would they have to take to produce the behavior you're seeing? How could they end up taking those particular values? What parts of the program and environment state are likely to vary, and how will they affect this routine?

Very often, you can answer those questions without getting bogged down in the call graph or having to try a huge number of cases. Sometimes you can not just reproduce the bug, but actually fix it.

• If function X is complaining about a "font not found" error, then it's presumably looking up some font name or other identifier in some font database. There probably aren't that many places that the font identifier can be coming from, and there's probably only one font database.

  If you can say, "well, the font should either be the default system font, or a font specified in the user profile", then you can make an intuitive leap. You know that everything uses the default system font all the time, so that path probably works... but maybe it's possible for a user profile to end up referring to a font that doesn't exist... but I know the code checks for that, and I can't set a bogus font on my own profile... but wait, what if the font gets deleted after the user picks it?

  Of course, there are lots of other possibilities^[2]. Maybe there's some weird corner case where the font database isn't available, or you're using the wrong one, or it's corrupted, or whatever. But it's unlikely to be something completely unrelated that's happening in some function in the call stack that doesn't use the font information at all.

• Or maybe function X is searching by font attributes instead of names. So where might it be getting extra constraints on the query?

• Or maybe function Y is blowing up trying to use a null value. The root cause is almost certainly that you made a poor choice of programming language, but you can't fix that now, so persevere. Usually you have the line of code where it choked, but say you don't. What values are in scope in Y that could be null? Well, it gets called with a font descriptor. Could that be null? Well, wait, the font lookup routine returns a null value if it can't find a font. So maybe we have a font not found error in disguise. So try thinking about it, for a limited time, as a font-not-found error.

• Or maybe "fnord!" is showing up randomly in the output. So where could it come from? First step: brute force. grep -ir 'fnord' src. Leave off the exclamation point at least to start. Punctuation tends to be quoted weirdly or added by code. If it's not there, is it in the binary? Is it in the database? Is it in the config file? Is it anywhere on the whole damned system? If not, that leaves what? Probably the network.

In the end, though, there's also a certain amount of pattern matching against experience. "Code that does X is usually structured like Y". "Does this thing have access to the files it needs?". "Programmers always forget about cases like Z". "I always forget about cases like W". "Weird daemon behavior that you can't reproduce interactively is always caused by SELinux".

Once you do reproduce the bug, you can always just switch to a strategy of brute force tracing everything that goes on anywhere near it, with a debugger, with printing or logging, with a system call tracer, or whatever. But, yeah, you've got to get it to happen if you want to trace it.

How does one "read the docs?"

It depends on what you're trying to find out.

I find I mostly use two kinds of documentation:

1. Architectural stuff. General material that explains the concepts running around in the code, the terminology, what objects exist, what their life cycles look like, etc. This sort of documentation is often either nonexistent, or so bloated and disorganized as to be useless for quick debugging, and maybe useless period. But if it exists and is any good, it's gold. If you're trying to educate yourself about something that you're going to use heavily, you may just have to slog through all of whatever's available.

2. API documentation, ideally with links to source code. I usually don't even skim this. I navigate it with keyword search. If you want to now how to frobnicate a blenk, then search for "frobnicate" and whatever synonyms you can come up with^[3]. If you're trying to debug a stack trace from a library routine, look up that routine and see what parameters it takes.

In my never especially humble opinion, "tutorials" are mostly wastes of time beyond the first couple of units, and it's unfortunate that people concentrate so much on them instead of writing decent architecture and concept documentation.

If it worked earlier today (and you didn't make any commits in the meanwhile), you may need to do something more like "binary search through your undo history."

If this is an issue consistently, it means you need to commit more often.

How do you learn to replicate bugs, when they happen inconsistently

I don't have definitive advice here, I think this is a hard problem no matter your skill level. You can do things in advance to make your program more debuggable, like better logging, and assertions so you catch the bug closer to the root cause.

A more general pattern to look for is some tool that can capture a particular run of the system in a reproducible/replayable manner. For a single program running locally, a coredump is already quite good, you can look at the whole state of your program just before the crash. (E.g. the whole stack trace, and all variables. This can already tell you a lot.) I have also heard great things about rr, supposedly it allows you to capture a whole execution and single step forwards and backwards.

For distributed systems, like web applications, the problem is even harder. I think I have seen some projects aiming to do the whole "reproducible execution" thing for distributed systems, but I don't know of any that I could recommend. In theory the problem should not be hard, just capture all inputs to the system, and since computers are deterministic, just replay the inputs. But in practice, given the complexity of our software stacks, often determinism is more of a pipe dream.

How does one "read the docs?"

Something something "how to build up a model of the entire stack."

I think these are closely related. I imagine my "model of the entire stack" like a scaffolding with some knowledge holes that can be filled in quickly if needed. You should not have any unknown-unknowns. If I notice that I need more fidelity in some area of my model, that's exactly the docs I read up on.

When reading docs, you can have different intentions. Maybe you are learning about something for the first time, and just want to get an overall understanding. Or maybe you already have the overall understanding, and are just looking for some very specific detail. Often documentation is also written to target one of those use-cases, you should be aware that (well) documented systems often have multiple of these. This is one model I came across that tries to categorize documentation (though I am not sure I subscribe to these exact 4 categories):

(from https://diataxis.fr/)

Getting back to the "model of the entire stack" thing, I think it's very important for how (I at least) approach computer systems. I think this article by Drew DeVault in particular was an important mindset-shift back when I read it. Some quotes:

Some people will shut down when they’re faced with a problem that requires them to dig into territory that they’re unfamiliar with. [...] Getting around in an unfamiliar repository can be a little intimidating, but do it enough times and it’ll become second nature. [...] written in unfamiliar programming languages or utilize even more unfamiliar libraries, don’t despair. All programming languages have a lot in common and huge numbers of resources are available online. Learning just enough to understand (and fix!) a particular problem is very possible

I now believe that being able to quickly jump into unfamiliar codebases and unfamiliar languages is a very important skill to have developed. This is also important because documentation is often lacking or non-existent, and the code is the "documentation".

Also, I feel like the "model of the entire stack" thing is a phase shift for debugging once you get there. Suddenly, you can be very confident about finding out the root cause of any (reproducible) bug in bounded time.

If at any point you notice that your unfamiliarity with some part of the system is impeding you in solving some problem, that's a sign to study that area in more detail. (I think this is easier to notice when debugging, but can be equally important when building new features. Sometimes your unfamiliarity with a certain area leads you to build a more complex solution than necessary, since you are unable to search paths that route through that area. A map analogy here would be you having a dark spot on your map, and noticing whether it's likely that between two points, there could be a shorter path through the dark area.)

I’m pretty mediocre as a dev but here goes:

Often debugging is hard because I dont want to invest time to understand the details or docs of whatever library I happen to be using. In that case a trick that often works is to keep my current project on pause for a moment, and build a toy example using that library. Keep increasing the complexity of the toy example and make it closer to the real bug, until something breaks. You also often notice that some of the code or params are redundant and do nothing.

These are very good 

When you are stuck, make explicit note of what feels difficult about the situation, and brainstorm ways of dealing with those difficulties.

Asking "This is impossible. Why exactly is it impossible?

These are very good 

When you are stuck, make explicit note of what feels difficult about the situation, and brainstorm ways of dealing with those difficulties.

Asking "This is impossible. Why exactly is it impossible?

walk up the stack trace

And start at the lowest level of your own code, but be willing to go into library code if needed.

How does one "read the docs?". Sometimes I ask how a senior dev figured something out, and they say "I read the documentation and it explained it." And I'm like "okay, duh. but... there's so much fucking documentation. I can't possibly be expected to read it all?"

You should read the docs like you would read a book; this provides outsized benefits precisely because most people won't do that. Source reasoning here:

https://aaronfrancis.com/2023/read-the-docs-like-a-book-2381721a

Do people just read really fast? I think they have some heuristics for figuring out what parts to read and how to skim, which maybe involves something like binary search and tracking-abstraction-borders. But something about this still feels opaque to me.

Even if you have poor heuristics, it's still may be worth it to google/open docs and walk obvious links. The point is not to have an algorithm that certainly finds everything relevant, but to try many things that may work.

How do you learn to replicate bugs, when they happen inconsistently in no discernable pattern? especially when the bug comes up, like, once every couple days or weeks, instead of once every 5 minutes.

You speed up time. Or more generally prepare an environment that increases reproduction frequency, like slow hardware/network or higher load. You spam clicks and interrupt every animation, because all bugs are about asynchronous things. You save state after reproduction, or better before and start from it. If all fails, you add logs/breakpoints to be ready next week. But usually you just look at code to figure out paths that may manifest as your bug and then try to reproduce promising paths.

I consider myself really good at debugging, but How does one "read the docs?" is something I never got good at? Notable devs like mitchellh skim the entire docs to get a vague idea of where stuff might be. This was never helpful for me. Docs seem to try to explain things in general whereas I think humans learn much faster from examples? We learn speech before grammar, we learn addition before number theory.

Searching for examples also biases your results to more commonly used functions. Function use is very Pareto distributed (small core of common functions and long tail of almost never used functions), docs hide this. Consider the libcurl function listing https://curl.se/libcurl/c/allfuncs.html. It hides the fact that  curl_easy_{init,setopt,perform,cleanup} is more important than everything else.

Finding good reference code is a skill in itself though. https://sourcegraph.com/search indexes ~all public code, and has great tools like lang:Go and -file:^vendor.  Keeping all relevant repos checked out in one folder you can ripgrep over also helps. tree-sitter and semgrep are well regarded, but I found them too distracting to be useful.