The 'programmers' are the human beings creating whatever advanced agent is being talked about.

The word 'programmer' isn't meant to imply that the AI is hardcoded as a fixed algorithm and then run; it's just the compact word we use to refer to the AI's creators. (E.g., the creation process could include explicitly coding algorithms, teaching the agent, exposing it to particular experiences, labeling its experiences for purposes of supervised learning, or any number of other nurturing processes extending over years.)

Many proposals for an AI's preference framework, or solving the value identification problem, require that the AI explicitly model the programmers and, even before then, figure out which objects inside the AI's beliefs about the environment are the programmers. These are respectively the programmer modeling and programmer identification problems.