This is an excellent question. Here's some of the things I consider personally important.

Regarding probability, I recently asked the question: Why is Bayesianism Important? I found this Slatestarcodex post to provide an excellent overview of thinking probabilistically, which seems way more important than almost any of the specific theorems.

I would include basic game theory - prisoner's dilemma, tragedy of the commons, multi-polar traps (see Meditations on Moloch for this later idea).

In terms of decision theory, there's the basic concept of expected utility, decreasing marginal utility, then the Inside/Outside views.

I think it's also important to understand the limits of rationality. I've written a post on this (pseudo-rationality), there's Barbarians vs. Bayesians and there's these two posts by Scott Alexander -  Seeing as a State and The Secret of Our Success. Thinking Fast and slow has already been mentioned.

The Map is Not the Territory revolutionised my understanding of philosophy and prevented me from ending up in stupid linguistic arguments. I'd suggest supplementing this by understanding how Conceptual Engineering avoids the plague of counterexample philosophy prevalent with conceptual engineering (Wittgenstein's conception of meanings as Family Resemblances is useful too - Eliezier talks about the cluster structure of thingspace).

Most normal people are far too ready to dismiss hypothetical situations. While if taken too far Making Beliefs Pay Rent can lead to a naïve kind of logical positivism, it is in general a good heuristic. Where Recursive Justification Hits Bottom argues for a kind of circular epistemology.

In terms of morality Torture vs. Dust Specks is a classic.

Pragmatically, there's the Pareto Principle (or 80/20 rule) and I'll also throw in my posts on Making Exceptions to General Rules and Emotions are not Beliefs.

In terms of understanding people better there's Inferential Distance, Mistake Theory vs. Conflict Theory, Contextualising vs. Decoupling Norms, The Least Convenient Possible World, Intellectual Turing Tests and Steelmanning/Principal of Charity.

There seems to be an increasingly broad agreement that meditation is really important and compliments rationality beautifully insofar as irrationality is more often a result of lack of control over our emotions, than lack of knowledge. But beyond this, it can provide extra introspective capacities and meditative practises like circling can allow us to relate better with humans.

One of my main philosophical disagreements with people here is that they often lean towards verificationism, while I don't believe that the universe has to play nice and so that often things will be true that we can't actually verify.

Hey, I've become interested in this field too recently. I've been listening to the Jim Rutt show which is pretty interesting, but I haven't dived into it in any real depth. I agree that it is something that we should be looking more into.

I won't pretend to be an expert on this topic, but my understanding of the differences is as follow:

• Systems theory tends to involve attempts to understand the overall system, while complex systems are much more likely to have emergent novel behaviour, so any models used need to be held more lightly/it's more likely that we have macro level trends that are there and we just don't know why
• Cybernetics is mostly about control systems (classic example is the thermostat). Feedback loops are an important part of systems theory, but they are just one particular tool.
• Regarding the breadth of applications, we can model dynamics in formal mathematical situations, then try to claim that similar dynamics occur in actual physical systems