FICO scores, by far the most popular credit score system, seem to be set up in a strange way. Their purpose is to measure the quality of a loan application, and yet their methodology seems quite suboptimal to do that.

From Patrick Mackenzie (Bits About Money): 

FICO scores are unreasonably effective. Many, many, many teams have thought “I bet I can get better loss rates if I supplement FICO scores with another data source”, and just about the only data sources for which that is actually true are illegal to use. 

And yet, if you look at the calculation of these scores, it basically is this:

Payment history (35%)
Amount owed (30%)
Length of credit history (15%)
New credit (10%)
Credit mix (10%)

Basically, this seems to just be

1. Have they paid loans back in the past (Payment History)
2. Will they pay loans back in the future
   1. Will they be unable to pay their loans back in the future (Amount owed, Length of credit history)
   2. Are they getting a lot of credit with the intention to not pay it back (Amount owed, Length of credit history, New credit, Credit mix)

1 is obvious, and likely why it is a ~third of the score. However, this is clearly not unreasonably effective. 

2a seems less effective than a bank worker directly looking at a customer's financial situation: these factors are basically a way to check someone's financial health without access to their income, as that is much messier.

2b seems to not be important enough to make up a significant portion of the score. The risk from credit not intended to be repaid is separate from risk accounted for via past loan delinquency base rates and future changes in financial situations, mostly as a separate, rare-but-consequential event. I don't think that adding the two tells you a lot about the person.

I think the most probable answers are the top-level bullets below, from most to least likely:

• These factors are finding something else
• The information about these scores is wrong
• FICO scores are calculated in a way other than what you would expect (adding the factors together, weighted by percent above and mapping to values between 300-850)
• FICO scores are less effective than I believe they are
• FICO scores measure something more intrinsic about a person (like general trustworthiness)
  • Why does credit age go into account here?
    • Is this just a legal way to account for older people being more trustworthy/having more to lose?
  • What about people goodharting on these
    • Is doing that actually evidence that they are trustworthy/more likely to repay loans?
    • Is 2b a way to counteract goodharting?
      • People still seem to be able to goodhart credit scores, however.
        • The methods used for this might only work on some portions, and goodharting could prove that you aren't untrustworthy because of a negative in this
          • Example: Credit Mix could mostly just be a way to measure how well the person understands credit, and goodharting your credit score shows you understand credit
• FICO scores just work by getting easy access to a lot of data
  • Why are they useful? Creditors already use a lot of data to make decisions about lending.

Essentially, FICO scores do not seem to be made with a special process. How can they be especially good data?