I haven’t looked into it, but it’s my impression that high viral loads come from virus replicating in your body, when you don’t yet have immunity. Once you have B-cell immunity from prior infection, virus has a much harder time replicating in your body, and you don’t get the high loads.

Like… if you’re expelling some viral particles with every breath, and the only viral particles you have are from your roommate, and you’re immune… then it shouldn’t take long to go through the particles you breathed in from the roommate, if there’s no internal replication going on in your body to generate new ones.

Hopefully someone else can give a better researched answer though!