Representation theory (which physicists and chemists sometimes just call Group theory) is the study of how groups act on vector spaces . A central idea in mathematics is that, because linear algebra is so easy (compared to other parts of math), we should always try to extract linear algebra from any situation we can, and representation theory is a standard tool to apply when extracting linear algebra from a situation involving a Group action.
In physics, vector spaces appear naturally as Hilbert spaces of quantum states associated to physical systems, and representations of groups on these Hilbert spaces appear naturally as symmetries of the system. The study of these representations is a fundamental organizing principle of modern physics; for example, it is behind Wigner's classification of particle types.