Imagine you're writing a Field Guide to Boats, and you want to know what you should include in your field guide. Barges? Rafts? These things?
You want something like a dictionary definition of boat. A descriptive definition that includes anything people commonly think of as a boat; an objective definition, because you're only writing one book, not a separate version for each reader.
Now imagine you're stranded on an island, and you open a bottle, and a genie comes out and gives you one wish, and you say, "I wish for a boat!", and the genie says, "Well, what's a boat?" And you know, because you've read stories, that the genie will take your definition of "boat" and try to screw you over. You'd better not read out the dictionary definition, or the genie will give you a toy boat, or a boat with a hole in it, or a kayak too small for you to fit into. You need a prescriptive, subjective definition of a thing that will transport you over water.
Too often I write a story, then read it and say, "Is that really a story?" It's easy to write something that fits existing theories of what a story is, and is emotionally powerful, yet seems to lack some integrity, relevance, or, hell, I might as well call it a "soul"--some magic stuff that makes a story satisfying.
I want to figure out what the magic stuff is so I can produce it consistently. So I've been blogging about the question, "What makes a narrative a story (to person P)?"
My blog quickly bogged down in unproductive debates about how I'm using the word "story". I'm trying to come up with a category definition that includes things I think are good dramatic stories, excludes things that I think are bad stories, and does not need to include poems, comedies, genre fiction, or meta-fiction. But people keep objecting that that isn't what the word "story" means.
True; it isn't. I'm trying to come up with a prescriptive, subjective definition of a dramatic story that says what a story ought to be in order to satisfy person P, exclusive of certain well-understood or highly-distinctive categories, so that I can write stories that satisfy a large number of people and also myself. But my readers are used to descriptive, objective definitions of story that describe things found in books, and can be used in public discourse without reference to a subject. It would be confusing if I said, "The 10 Best Dinosaur Porn Stories anthology didn't have any stories in it."
I thought it was obvious from context that I was looking for a prescriptive, subjective definition of story, but it was not. At all.
(Also, dinosaur porn is a thing now.)
I've seen other discussions deadlock because people didn't realize that some of them were using prescriptive definitions, some descriptive, some objective, some subjective.
An atheist makes observations about religion, using examples from Unitarian Universalism, Christianity, and Scientology. A Unitarian responds that Scientology isn't a real religion, because its founder and its chief proponents are cynical, deceptive, and uncharitable. A Christian responds that neither is Unitarian Universalism a real religion, since it's different for every UUist. The atheist thinks the idea of a "real" religion doesn't even make sense; religions are what other people point at when they say "religion" (descriptive, objective).
The three people are at an impasse, because the Christian framework requires a prescriptive, objective definition of religion, the Unitarian requires a prescriptive, subjective definition, and the atheist requires a descriptive definition. You could even say that UU is defined as the belief that religion must be defined prescriptively and subjectively.
And then there's the Hopi Indian who told me, "As a Hopi I believe the old myths; as a modern person I do not." He had a descriptive, subjective definition of "belief".
Perhaps people pay so little attention to these distinctions because Western thought is heavily Platonist, and Platonism says that there are no subjective definitions, and there is no distinction between prescriptive and descriptive definitions--the only world we can describe conforms to prescriptive, objective definitions.
We run into even more problems when topics are complex enough that we must ask of subjective definitions, "The subjectivity of whom?" John Searle's famous Chinese-room argument is a sleight-of-hand that exploits the ambiguity of natural language to swap different subjectivities into the scenario at different times. Keep your eye on the subjectivity, and it falls apart. You could say that once you've accepted the framework of utility theory, ethics is primarily about whose subjectivity to use when optimizing utility.
To relate this to AI ethics (not my main point, but I said yesterday that giving a motivational example is good):
Three people are discussing ethics. One believes in natural law, which is prescriptive and objective. Another is an evolutionary biologist who believes that a species plus an environment leads to a set of behaviors that maximize reproductive fitness. That's a prescriptive, subjective definition of ethics, but the subjectivity is that of genes, not of "individuals", whatever they are. Another is trying to build a "Friendly AI". According to FAI theory, he needs a not-completely-subjective definition, since he's not supposed to build his own subjective definition into the AI and its subjectivity is not pre-determined. According to CEV theory, he needs it to be descriptive, since CEV asks you to extrapolate from observed human morality rather than try to derive ethics from first principles. So he wants a descriptive, semi-objective definition.
Speaking more strictly, I think that CEV attempts to take a descriptive, subjective definition that covers all subjects S where S is the rational, conscious part of a human mind, and extrapolate it to a descriptive definition for all subjects S where S is the computational closure of the rational and conscious parts of any human mind (which has problems due to the requirement of infinite computational power, but let's set that aside). We usually use the term "objective definition" to mean a definition used by all humans, because non-humans on Earth don't talk. But in CEV this distinction between objective and subjectivity(HUMANS) becomes very important, as we must consider intelligences unrelated (evolutionarily) to humans.
An FAIer defending the FAI+CEV program may begin with the assumption that he's looking for a descriptive, subjectivity(rational_part(conscious_mind(HUMANS))) ethics that doesn't contradict ethics(descriptive, subjectivity(computational_closure(rational_part(conscious_mind(HUMANS))))). He then tries to show how you could design a CEV algorithm that will construct such a system of ethics.
Whether or not a CEV specification can be devised that can achieve its goal isn't of much interest until you address the prior contentious points of whether we want ethics(descriptive, subjectivity(computational_closure(rational_part(conscious_mind(HUMANS))))), and whether that is even a coherent concept. There are many huge logical leaps in asserting that that's a coherent subjective perspective, and philosophical leaps in claiming that any of us, individually, should care about it.