One particularly amusing bug I was involved with was with an early version of the content recommendation engine at the company I worked at (this is used by websites to recommend related content on the website, such as related videos, articles, etc.). One of the customers for the recommendation engine was a music video service, and we/they noticed that One Direction's song called Infinity was showing up at the top of our recommendations a little too often. (I think this was triggered by the release of another One Direction song bringing the Infinity song into circulation, but I don't remember what that other song was).

It turned out this was due to a bug where we were taking a dot product of feature values with feature weights, where the feature value was being cast from a string to a numeric, with a fallback to zero if it was non-numeric, and then multiplied by the feature weight. For the "song title" feature, the feature weight was zero, and the feature value was anyway non-numeric, but even if it were numeric, it shouldn't matter, because anything times zero is ... zero, right? But the programming language in question treated "Infinity" as a numeric value, and it defined Infinity * 0 to be NaN (not a number) [ETA: A colleague who was part of the discovery process highlights that this behavior is in fact part of the IEEE 754 standard, so it would hold even for other programming languages that were compliant with the standard]. And NaN + anything would still be NaN, so the dot product would be NaN. And the way the sorting worked, NaN would always rank on top, so whenever the song got considered for recommendation it would rank on top.

This reminded me of a bug I spent weeks figuring out, at the beginning of my career. Not sure if something like this would qualify, and I do not have the code anyway.

I wrote a relatively simple code that called a C library produced at the same company. Other people have used the same library for years without any issues. My code worked correctly on my local machine; worked correctly during the testing; and when deployed to the production server, it worked correctly... for about an hour... and then it stopped working.

I had no idea what to do about this. I was an inexperienced junior programmer; I didn't have a direct access to the production machine and there was nothing in the logs; and I could not reproduce the bug locally and neither could the tester. No one else had any problem using the library, and I couldn't see anything wrong in my code.

About a month later, I figured out...

...that at some moment, the library generated a temporary file, in the system temporary directory...

...the temporary file had a name generated randomly...

...the library even checked for the (astronomically unlikely) possibility that a file with given name might already exist, in which case it would generate another random name and check again (up to 100 times, and then it would give up, because potentially infinite loops were not allowed by our strict security policy).

Can you guess the problem now?

If anyone wants to reproduce this in Python and collect the reward, feel free to do so.

I just had a surprisingly annoying version of a very mundane bug. I was working in Javascript and I had some code that read some parameters from the URL and then did a bunch of math. I had translated the math directly from a different codebase so I was absolutely sure it should be right; yet I was getting the wrong answer. I console.logged the inputs and intermediate values and was totally flummoxed because all the inputs looked clearly right, until at some point a totally nonsense value was produced from one equation.

Of course, the inputs and intermediate values were strings that I forgot to parse into Javascript numbers, so everything looked perfect until finally it plugged them into my equation, which silently did string operations instead of numeric operations, producing an apparently absurd result. But it took a good 20 minutes of me plus my coworker staring at these 20 lines of code and the log outputs until I figured it out.

Maybe looking at the code for games would make sense because in games we optimize bugs way very differently than for other software. Usually, when there is a bug in a game, it's not critical. Many developers only do the kinds of testing where you play the game and then see if there are any bugs that you run into that need to be fixed.

I expect this overall approach to developing software would lead to subtle bugs that are hard to fix.

This is partly because of the complexity of games, but also because bugs that are not critical usually will not get fixed. You start to build upon these bugs, possibly creating a very rigid structure such that anything within the structure, including the pre-existing bugs, will be tough to fix later on.

However, this might not be what you're looking for, especially when we're talking about games written in engines like Unity, because then the structure of the program is extremely different from something like a Python module, and to properly evaluate it, you would need to look at and understand some video stream,

However, maybe this is still interesting because it does present a sort of very, very tough-to-crack kind of programming challenge, really because the programmers who write the code mainly look at the video stream to evaluate it, and therefore, the overall problem of engaging with the program and realizing what is even wrong is a lot harder. At least some of the time.

I do have some pretty horrible spaghetti code mess games written in Unity that have not been posted publicly but again, I would expect that this is not that useful to you therefore I will not submit it unless you tell me otherwise.

Another thing to consider is to look at is code for shaders, which has similar properties to what I outlined above, but in a more accessible way and is a lot more self-contained and smaller in scope (also has the same problems you potentially need to look at a video stream to evaluate if it works).

Some examples of bugs that were particularly troublesome on a recent project.

1. in the MIPS backend for the LLVM conpiler there is one point where it ought to be checking whether the target cpu is 32 bit or 64 bit. Instead, it checks if the MIPS version number is mips64. Problem; there were 64 bit MIPS versions before mips64, e.g, mips4, so the check is wrong. Obvious when you see the line of code, but days of tracing though thousands and thousands of lines of code till you  it.
2. with a particularvversion of freebsd on MIPS, it works fine on single core but the console dies on multi core. The serial line interrupt is routed to one of the cores. On receiving an interupt, the oscdisables the interrupt and puts handling the interrupt on a scheduling queue. when the task is taken off the queue, the interrupt is handoed and then re-enabled. Problem: last step might be scheduled to run on a different core. If that happens, interrupt remains disabled on the core that receives the interrupt, and enabled on a core that never recieves it. Console output dies.