I think it's a good question. A few of my thoughts:

Goals seem to be a fundamental part of how human minds works. It's impossible to move your arm without having a goal of where you want it to move.

If you switch off having a goal for the movement in hypnosis and tell someone to try to move their arm, they tense up the muscles of the arm but don't actually move the arm.

I'm underweight. I tried a lot of interventions and measured my weight every day. One month I added 800 kcal of maltrodextrin to my tea every day. However doing it like that didn't work well.

Then I decided I have enough. I simple make a decision to gain weight and let the body figure out what needs to be done on it's own. That worked till I was occupied with other stuff. The fine control didn't help, on the other hand the intention helped a lot.

A strong intention organizes all the mental processes. I'm more likely to follow an impulse of hunger. My body puts out the necessary hormones to put on weight.

Areas where there are lot of conflicting goals produce a drain of mental energy.

Having goals that you achieve creates a feeling of agentship. It makes you confident in achieving other things in the future.

Like anything, it's a cost-benefit. Here's my impressions.

Benefits:

• It feels good to accomplish the goal. (If I make a goal to run every day for a month and I accomplish it, it'd feel good. If I had otherwise happened to run every day for a month, I wouldn't experience that feeling of accomplishment at the end of it (in the same way).
• It motivates people. What I describe above is part of "the carrot", but having goals also acts as "a stick". Because when you know you have a goal to do something, you're motivated to avoid the unpleasantness of not having achieved your goal.
• Accomplishing goals boosts peoples confidence. When you are successful, you think, "I accomplished a Thing". This sort of explicit realization might not happen if you hadn't made it an explicit goal. With the realization might come feelings of, "I'm the sort of person who could accomplish things in general" or "I'm the kind of person who could accomplish similar things to the thing I just accomplished".
• As a form of precommitment, it may help prevent you from rationalizing. Without a goal, it may be easy to say, "eh, maybe I shouldn't do this". But this is often a rationalization, short-sighted, fear-driven, impulse-driven etc. By setting a goal and sticking to it, you're sort of saying, "Hey future self - I declare that you don't have the authority to come up with rationalizations. You're not allowed to decide not to do this.".
• (Each of these could be broken down further. Ex. you could make the distinction between feeling good right after accomplishing the goal of running every day for a month, and the feeling of looking back on that accomplishment 5 years in the future.)

Costs:

• It's not "flexible". What if you set a goal for yourself, but subsequently realize that the goal isn't "what's best for you"? Like what if you realize that you've been seeking lost purposes or something? (There's a trade-off with the fourth bullet point of Benefits though)
• It may be stressful. Having a list in your head of "stuff to do" is often stressful. Alternatively, you could just throw out that to-do list and say, "I'm just going to decide on the fly what's best for me to do; there's nothing that I have to do".
• It also may be stressful in the sense that you are worried that you won't achieve it.
• If you fail at your goal...
  • It may make you unhappy (because failing is unpleasant).
  • You may lose confidence and think, "I'm the sort of person that fails".
  • You'll probably feel less motivated to pursue goals in the future.

Personally, I really don't like the fact that goals "remove flexibility". The way I see it, what I should do depends on expected utility calculations, and these calculations change as I acquire new information and make progress in my analyses. I don't like the idea of putting myself in a situation where I'm not allowed to pursue the option that I calculate has the highest expected utility.

But I do find them moderately fun and motivating, and so I set "flexible goals" for myself.

I should also mention that one of the most important things that happened in my life involved setting an absolute goal. In 8th grade I made it a goal to make the high school basketball team the next year. I was pretty bad at the time and conventional wisdom said I had absolutely no shot. But I was extremely committed and I did it, and this made me happy and gave me a lot of confidence (sort of; it's more complicated).

Re: generally applicable advice - I've never been a fan of it (in this context, and also more generally). In the context of goals, I sense that people are different enough such that it's pretty hard to find generally applicable advice. Ie. they weigh the costs and benefits in the bullet points differently.

I'm not sure to what extent this is true though. I'm not particularly familiar with the literature, and I haven't thought about it too hard. I just sense that advice is a lot less generally applicable than "people" make it out to be.

Setting a goal helps clarify thought process and planning; it forces you to step back a bit and look at the work to be done, and the outcome, from a slightly different viewpoint. It also helps you maintain focus on driving toward a result, and gives you the satisfaction of accomplishment when (if) you reach the goal.