Any references or studies behind this? My intuition (for which I know of no studies, so is just another option from yours) is different: most people are unwilling and/or unable to do the work of science/engineering - they are drawn to "do what I want" solutions rather than "do what I very precisely describe" solutions.
It's not about internal capabilities vs external tools. It's about what level of understanding and attention-to-detail is required to use it.
It could be both -- the hard "magic" (learning quantum physics and constructing your own device) is too hard for most people, and the soft "magic" (using a standard device bought in the nearest supermarket) feels too impersonal, and also doesn't make you special.
So perhaps the desire for magic is the desire for awesome things to be (1) rare, but (2) relatively simple for the protagonist, and (3) feel natural after mastering them. A world where at most 10 years of studying somehow makes you able to to do all kinds of awesome stuff semi-automatically, while most of the population somehow can't do the same.
(Uncharitably, we could say that it is a desire to achieve extremely high prestige relatively cheaply.)
Also, the magic is supposed to be useful for personal purposes. Suppose you are an expert on quantum physics -- does it allow you to cast fireballs in self-defense, build a fortress on the top of the mountain, levitate, turn your neighbors into zombies, become invisible, or see through walls? Your magical equivalent in the parallel world can do most of this.
It can absolutely be both (or all three - I take "being special" as distinct from "your own power"), in different mixes for different people.
I'm not sure about the "useful for personal purposes" part. A lot (but not all) fantasy wizards pay a fairly high social cost for their erudition, and it's not clear that all that studying and experimenting helps them nearly as much as just getting rich and hiring an army would.
All that said, there's a limit to what we should learn from fiction.
I concur on "do what I want" as the distinction. Magic is teleological: it produces a certain effect, whereas science is about the cause. Magic "just works." You do not need to know how the magic words produce light or fire, they just do. The great usual magical dream is about having massive power under your control, not whether you are the direct cause. Wishes are the archetypal example. You wish for something and it happens. Done. A genie is neither you nor a tool you control, just a massive source of power bound to your wishes.
Magical horror stories are about wishing for effects and either not liking the cause or not specifying the effect properly, like the evil genie or the monkey's paw. Science horror stories are about starting a cause without realizing its effects.
There are plenty of science-like magical stories, where they delve into the rules of magic and effects from causes. And don't those read a lot more like science fiction, whereas a space opera like Star Wars has the trappings of science fiction but Jedi are just space wizards that produce effects by willing them.
I think there can be something else to it; Magic, in most stories, requires tremendously less infrastructure. It takes some years of study and a short chant, say, to create a fireball.
To make a flamethrower takes some years of study, a couple of mines, thousands of people, manufacturing capability, etc. There's a lot more personal effectiveness in magic.
Disclaimer: I'm a mechanical engineer that spends most of my time on software these days. I can make all sorts of intersesting things, but the appeal of being an archwizard is definitely there.
It is an annoying thing about my mind that I can remember something without knowing where it as from. If I can located my original sources I'll will add them. However I do observe that people are drawn to the idea of doing something themselves out of their own abilities over using outside means such as tools, money or political power. And I predict that if people could see science not just in terms of inventions but as a way to become stronger as a human being, then more people would be drawn to it.
I might just being falling into the typical mind fallacy but that is still my prediction. I would like to expand my knowledge on this matter and find a way to test this prediction, but I don't know how to do it right now as things are
I have sometimes mused that accumulating political power (or generally being able to socially engineer) is the closest to magic that we have in the real world. It's the force multiplier that magic is used for in fiction by a single protagonist. Most people who want magic also do not follow political careers. Of course, this is only a musing because there are lots of differences. No matter how much power you accumulate you are still beholden to someone or something, so if independence is a big part of your magical power fantasy then it won't help.
I would argue that the closest real-world analogue is computer hacking. It is a rare ability, but it can bestow a large amount of power on an individual who puts in enough effort and skill. Like magic, it requires almost no help from anyone else. The infrastructure has to be there, but since the infrastructure isn't designed to allow hacking, having the infrastructure doesn't make the ability available to everyone who can pay (like, say, airplanes). If you look at the more fantasy-style sci-fi, science is often treated like magic--one smart scientist can do all sorts of cool stuff on their own. But it's never plausible. With hacking, that romanticization isn't nearly as far from reality.
Similarly I have mused that the closes thing we have to magic in the real world is the ability to forecast.
The closest thing we have to magic in real world is advanced technology.
Move your mind a hundred years back and look at your phone.
This reminds me a little of something like Marxist's comment of the alienation of labour. It also reminds me of Jacque Ellul's remarks that technology make responsibility impossible.
Back in 2008 Eliezer wrote the following post http://lesswrong.com/lw/ou/if_you_demand_magic_magic_wont_help/ In it he said the following
"Born into a world of science, they did not become scientists. What makes them think that, in a world of magic, they would act any differently?" However, he seems to misunderstand. I do not believe that people seek magic or other fantasies for being different and on the other side of the fence. But rather, because it fills a need that most sciences don't give people.
In fantasy when people imagine the kind of things they want, I tend to see the following things
flying under you own power vs airplanes
powerful melee weapons vs guns,
supernatural strength and speed vs tools and vehicles,
magic vs tools.
What these share in common is they are your own power vs a outside power. People feel a stronger connection to their actions when they are doing it with their hands or with a tool that uses their own strength. Hitting a light switch to turn on a light doesn't feel like something you did. Inventing the light bulb makes you feel what you did something, but the result still doesn't feel like a part of yourself. Likewise, a sword feels like a part of you in a way a gun doesn't. Cars do better than other vehicles because you become attached to them mentally.
Magic fulfills this need to have your ability be your own on a emotional level. A light spell is your own light and I think people would feel the same way if they had a biological ability to emit light, if you could fly on your own, with your own wings then people might stop thinking about broomsticks. Being a cyborg gives similar benefits. The machinery is a part of you so you feel like your actually strong. Like you actually can create fireballs, leaps 10 stories or walk through a fire unharmed while giving one liners.
I am not sure how to make people feel the same way when it comes to current science and technology. We can make people feel better in life in general if we started to promote blue collar work again, but that is leaves the matter that people aren't pursuing science as much as I wish. I would like to bring this into discussion before trying to propose a solution as any solution I can comeup up with now is "in the future" what can we do right now to handle this without cyborgs.
Please discuss.