If you ask a Software Architect which database technology you should use
Well, it depends on the architect, I suppose. I would say "write your application in a modular way so that it doesn't rely on specific features of the underlying database, and if you need to make exceptions to this principle, make sure the relevant code sections and dependencies are well highlighted and documented."
I agree that as much as possible you want to make things modular, but good data processing code for a relational database is going to be really different than good code for a distributed key-value store or a mapreduce+cache setup.
Good call. This is a great example of how sometimes experts can even point out that you're thinking at the wrong level of abstraction, and provide a better one.
When learning comething new from IT, I make my first steps with tutorials, but then I try to read the information that the experts use. For example for a new programming language, I try to read the official API reference. Or for something about HTML / CSS / XML / XSLT, I try to read the W3 specification.
Even if I am not able to understand the new thing yet, I do the tutorials, but I try to find the words from tutorials in the reference. Soon it "clicks" and I see the correspondence between how the tutorial says it and how the reference says it.
The idea is that the reference was made by experts, while the tutorials could be made by anyone, so it is better to be able to read the reference when possible. Also the reference contains everthing, while tutorials are limited, so using the reference gives more freedom.
Obviously, being an expert in other programming languages helps me a lot here, because many concepts are the same. But if I were teaching students, after the very basics I would try to teach them the reference-reading skills, and from that point on they could explore on their own.
There are a couple chapters in there on the subject, but it's probably not the best book specifically for that subject. I haven't read it yet, but "Working Minds: A Practitioner's Guide to Cognitive Task Analysis " looks pretty good. One of the people behind the method wrote several books for a general audience and one of them, "The Power of Intuition" (not what it sounds like) has a few tips on how to learn from experts:
*Probe for specific incidents and stories. This is not the same thing as listening to war stories. It means selecting incidents where intuition was needed, and expertise was challenged, and then digging into the details.
*Ask about cues and patterns. Try to find out what the expert was noticing while making sense of the situation. You want to uncover types of discriminations that the expert has learned to make, types of patterns the expert has learned to recognize. The decision-making critique can suggest lines of questioning.
There's more, but I don't want to quote too much... One of the other tips is to ask how a novice would approach things vs how they would. He also recommends avoiding asking for general theories, the experts may not really be able to describe how/what they do without having a story to guide them.
Somewhat. It's a comprehensive study with a lot of good facts on expertise. But the the method is something I came up with and haven't seen elsewhere. So you may have to spend a significant chunk of time thinking about the implications of these facts in order to derive something useful.
To what extent does software development expertise transcend the specific tools one is using? What general skills can I develop that will make me a better programmer even 20 years from now when Ruby on Rails is looked at the way COBOL is looked at today?
Another important aspect to take into account here is cross-domain similarities; those stories of experts from one field chancing across a principle or development in another field and having a revelation about what to do in their own field, often with great results. Wish I would remember a powerful example of this off-hand.
The key difference between experts and beginners is the quality of their abstractions. Masters of a field mentally organize information in a way that's relevant to the tasks at hand. Amateurs may know as many facts and details as experts but group them in haphazard or irrelevant ways.
For example, experienced Bridge players group cards by suit, then number. They place the most importance on the face cards and work down. Bridge amateurs group solely by number and place equal importance on all numbers. Professional firemen group fires by how the fire was started and how fast it’s spreading-features they use to contain the fire. Novices group fires by brightness and color. Both have the same information, but the firemen hone in on the useful details faster.1
Learn abstractions from masters. If you ask a Software Architect which database technology you should use, circumstances will eventually change and you'll need to ask them again and pay them again. But if you ask the Architect to teach you how to choose a database then you can adapt to changing circumstances. Ideally you should emerge with a clear set of rules-something like a flow-chart for that decision. A good example is this article on whether you should use hadoop. Clear criteria let you make a high-quality decision by focusing on the relevant details.
After talking to the expert you can write up the flow-chart or criteria and send it to them to get their opinion. This ensures you understood what the expert was trying to say, and lets you get additional details they might add. Most importantly it gives them something valuable to share with people seeking similar advice, so you're able to add value to their lives as a thank you for their advice.
Caveats to this method: