You do realize that there's >100 years of research on this topic about human cognitive abilities/intelligence? Reading this literature requires some understanding of statistics, but you would do well to read Stuart Ritchie's, Deary's or Haier's recent book length summaries of the area. Arthur Jensen's book is the best, but it's not an easy read.
Multiple selection is discussed in the animal breeding literature. See e.g. this review.
Samorè, A. B., & Fontanesi, L. (2016). Genomic selection in pigs: state of the art and perspectives. Italian Journal of Animal Science, 15(2), 211–232. doi.org/10.1080/1828051X.2016.1172034
Sometimes the traits selected for are negatively genetically correlated. This slows down the process, but does not make it impossible unless the genetic correlation is -1.00. For humans, most of the traits we want seem to be positively related, with a few exceptions. Sometimes bipolar and IQ have positive relationships, which may be undesirable. Bipolar is associated with creativity however, so perhaps it's not entirely bad. A larger problem is the negative genetic correlation between fertility and IQ. There's also myopia and IQ.
Overall a sensible introduction. Turkheimer is very leftist by the field's standards, you may want to try out some others for opposite political slant or no slant. Not all authors color their work so heavily as he does.
You make a mistake in your terminology. Epistasis is not the same as nonlinearnity. Linearity stands in contrast to nonlinearity, which is called dominance. This is when a subject's phenotype does not change in a linear fashion from changing the alleles. So e.g., the change from TT to AT to AA does not have equal steps. This is common with severe genetic disorders, where having one copy usually does nothing. This is because a correct version of the protein (say) is still made half the time, and this is enough. Epistasis is gene-gene interactions between loci. In other words, there is no clean effect of changing from TT to AT because the effect depends on another locus where e.g. GG, or GC may be present.
There have been many papers looking for dominance and epistasis, but little has been found. EA4 tested across the genome for dominance and found nothing. Not due to low power. Epistasis generally causes MZ to be more than 2DZ, which is not commonly seen. Usually, the bias from assortative mating is stronger than any unmodelled epistasis or dominance.