This is a special post for quick takes by frmsaul. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
As a student at the Technion, engineering exams evaluated both my capabilities and my ethics. What does it mean? It's all downstream of this story:
In 1961, the late Professor Haim Hanani was appointed as the vice president of the Technion. Once appointed, Hanani proposed to start teaching also humanities. The other professors did not agree. They claimed that there’s barely enough time to teach "hard" sciences.
Hanani wanted to prove his colleagues wrong. He gathered a hundred students, and gave them an exam with one question: what technical information do you need to plan a pipeline to transport blood from Ashdod to Eilat? The two cities are located 250km apart.
The students started working on a solution right away. Using drawing boards and slide rules, they suggested to measure the topographical situation along the route, check the pipe layout, and test the corrosion resistance.
When they finished and submitted the test, Professor Hanani announced that they all failed: "I did not ask to test your ability to plan a blood pipeline, but to examine your moral sensitivity. None of you asked whose blood will flow through the pipes, or who is asking to build it in the first place".
Nowadays, Technion professors will sometimes hide ethically loaded questions in exams, my responsibility as a student was to find them and point them out instead of "just following orders" and answering them verbatim. So every time I faced a question on a problem-set or an exam, I spent a few seconds trying to decide if it's a "blood pipeline" question or not. "Does this linked list represent a blood pipeline? Could this linear program be used to optimize the intake in a concentration camp?". I don't think I was ever actually presented with such a problem[1], but the blood pipeline was always in the back of my head.
In the language of ai alignment, you can say I was evaluation aware, maybe even evaluation paranoid. After leaving school, this habit didn't really stick. I haven't thought about the blood pipeline in years.
If I was, I didn't notice
If he had asked how long it would take Superman to fly between those two cities giving a starting speed and acceleration, etc., the proper response would be to calculate it, not say "Superman doesn't exist, you must be testing to see if I can recognize unrealistic scenarios." If the professor had then said "what if you had a clueless boss who gave you a job that actually assumed that Superman is real? You could bankrupt the business by your failure to question the order!", it would be obvious that he was trolling.
Abstracting away the details may be wrong if you are actually being asked to build a blood pipeline, but correct in a test.