Developing skills related to AI puts you in a better position to make AI go well. At least for me, this outweighs the other concerns that you've mentioned.
Note: This doesn't mean that you should take a job that advances fundamental AI capabilities. This would probably be net-negative as things are already moving far too fast for society to adapt. But it sounds like you're more considering jobs related to AI applications, so I'd say to go for it.
I think that there are two questions one could ask here:
Is this job bad for x-risk reasons? I would say that the answer to this is "probably not" - if you're not pushing the frontier but are only commercialising already available technology, your contribution to x-risk is negligible at best. Maybe you're very slightly adding to the generative AI hype, but that ship's somewhat sailed at this point.
Is this job bad for other reasons? That seems like something you'd have to answer for yourself based on the particulars of the job. It also involves some philosophical/political priors that are probably pretty specific to you. Like - is automating away jobs good most of the time? Argument for yes - it frees up people to do other work, it advances the amount of stuff society can do in general. Argument for no - it takes away people's jobs, disrupts lives, some people can't adapt to the change.
I'll avoid giving my personal answer to the above, since I don't want to bias you. I think you should ask how you feel about this category of thing in general, and then decide how picky or not you should be about these AI jobs based on that category of thing. If they're mostly good, you can just avoid particularly scummy fields and other than that, go for it. If they're mostly bad, you shouldn't take one unless you have a particularly ethical area you can contribute to.
I suppose I'm mostly also looking for aspects of this I might have overlooked, or inside perspective about any details from someone who has relevant experience. I think I tend to err a bit on caution on things but ultimately I believe that "staying pure" is rarely a road to doing good (at most it's a road to not doing bad, but that's relatively easy if you just do nothing at all). Some of the problems with automation would have applied to many of the previous rounds of it, and those ultimately came out mostly good, I think, but also it somehow feels This Time It's Different (but then again, I do tend to skew towards pessimism and seeing all the possible ways things can go wrong...).
It's definitely overhyped. I hesitate to call it a bubble - it's more like the normal software business model with a new cover. Tons of projects and startups with pretty tenuous business models and improbable grand visions, most of which will peter out after a few years. But that has been going on for decades, and will likely continue until true AI makes it all irrelevant.
Most of these jobs are less interesting, and less impactful than they claim. Which makes the ethical considerations far less important. My advice is to focus on the day-to-day experience and what you can learn there. Pick one where you like the people, and get to actually build something rather than just rearranging existing crap.
it's more like the normal software business model with a new cover
True enough, though it's also the fact that these projects seem to have almost entirely displaced everything else that makes me suspect we're almost in bubble regime. VCs just throwing money at anything that involves AI.
Most of these jobs are less interesting, and less impactful than they claim.
Well, I mean, they could be somewhat impactful in expectation. One out of a hundred might become big, and you don't know which (in fact, 1% I suspect would be a good success rate...).
I've been on the lookout for new jobs recently and one thing I have noticed is that the market seems flooded with ads for AI-related jobs. What I mean is not work on building models (or aligning them, alas), but rather, work on building applications using generative AI or other advances to make new software products. My impression of this is that first, there's probably something of a bubble, because I doubt many of these ideas can deliver on their promises, especially as they rely so heavily on still pretty unreliable LLMs and such. And second, that while the jobs are well paid and sound fun, I'm not sure how I feel about them. These jobs all essentially aim at automating away other jobs, one way or another. That is a good thing only insofar as various other things happen, and depending on the specific job and quality of the work - a good automated GP for diagnosis would probably do a lot of good, but a rushed one might be net negative, and automating creative work is IMO just the wrong road to go down in general if we want good AI futures.
What are your intuitions about this? Which kinds of AI jobs do you consider having more potential for overall positive/negative value for society?