Proposing that sorting by personality similarity would make pleasant, cohesive families is making an assumption about human nature, which I wish like hell were true, but it's not: It assumes that if only everyone were matched with the right people that "clicked" with their personality, they'd all find their role in happy family units and we'd all live happily ever after. And that's probably true for many or even most people, people who could be miserable and a drag on everyone if they were surrounded by the wrong partners, but turn into happy and contributing family members when they're placed with the right ones.
But I believe the unfortunate reality is that society is full of people who are just plain shitty to be around. No matter the context, no matter who you placed them with, they'd make life miserable for everyone around them. Pick a reason--endless lying, arguing, complaining, hypochondria, paranoia, toxic jealousy, plenty of people have some or all of those traits that make one a bad family member to almost anyone. I'm drawing from personal experience here, I can't cite exact personality traits or evidence to back up my claim, but I think it's a pretty safe bet to say high neuroticism is a factor. Unless you're incredibly fortunate, you probably know people in your own extended family whose company you wouldn't wish on your worst enemy. I think that you could have the most advanced sorting AI in the world to try to find the perfect matches for many people and the family units they were in would still be worse off for their presence.
One advantage--at least in terms of seeing that even the most unpleasant and burdensome members of society are cared for--of the "luck-of-the-draw" family system is that these people are placed within a support system of others who are socially obligated towards them. Again, your experience may vary but I personally know a good handful of people that would probably be living under a bridge without a friend in the world if their family members didn't feel obligated to keep taking them in and giving them support despite their abrasive personality and long track record of terrible behavior.
Some of this you could chalk up to mental illness, and your idea might be a little more feasible if we took that out of the equation with better AI-driven medicine and institutions, but unless you're going to add "Being a general asshole" to the DSM, I think that leaves a lot of people who wouldn't be a good fit anywhere.
This is one advantage of polyamory. It’s not discussed much, but it’s the kind of thing you see when someone gets hurt. When my girlfriend broke her leg, me and her other partner were able to coordinate and help her more than either of us could alone.
One thing is that some of the 'family failing' cases actually feel like a huge dark side to the 'being stuck with them' aspect (there's values drift from immersing in media/online nowadays that can make people feel less connected, but also in cases of abuse kept secret, not liking being legally trapped for years, feeling held back far behind their peers if their family is late to important events).
I would also say online longtime friends of a community or group (or streamers) that meet up can also fit the idea of another institution like that, but it's heavily community dependent, I could say in theory there are probably some who do this and then permanently live nearby each other or in the same house but this is not common.
Family is extremely important to a lot of people. It is also an institution we have had as long as we have existed.
If we wanted to, how hard would it be to create another institution with similar virtues? (Not to replace families in general, but to have more of that value, or a backup for when family fails, as it does for many people.)
People often consider close friends to to be their ‘chosen family’, but I think the inability to choose is relevant to some of the virtue of real families: you are stuck with each other almost no matter what, and you are forced to be on good
A rough method I could imagine: take a large set of people and use AI to sort them by some measure of personality similarity and physical proximity into groups of 4-10, across different ages. Give each group substantial incentive to interact and be on good terms with each other, for instance free use of local amenities if they show up together. (I’m imagining this is brought about by a national government, but alternately a large group of people could decide to do this with each other.)
This plan is probably missing something like the meaning and loyalty-to-ones-blood aspects of family. Though I could imagine the right personality assessment system might change that, if people felt they were being put in touch with essentially their platonic soul-mates.