Hope1234
Message
I am currently a graduate student (masters in EE) at Santa Clara University studying DSP and AI. Before that, I worked as an FPGA engineer for a few years then switched professions to patent examining at the USPTO for 11 years. I started out examining...
6
5
Thank you for the work, it's great! Appendix G is informative.
I had a question about early-layer divergence. Fig 17 shows the expert jacobians disagreeing in direction at earlier layers and agreeing later on. Figure 16 shows lower effective rank early. This could be what you would see if the Jacobians are partly rotated relative to each other and averaging would cause a loss of information (incoherent combining, in signal processing).
Did you try aligning the experts before averaging? If I were going to do it, I would try polar-decomposing each expert Jaco... (read more)