Interesting that there seems to be very little research in preventing tolerance effects. Maybe there is an opportunity there.
I'd recommend checking out mild nootropics used by other cultures on a daily basis. Slow-burning 'highs', like the kind you get from South American maté, seem easier for my body to handle than the steep and sharp coffee buzz. There's also the obligatory meditation plug, but it's probably pretty reasonable to assume that anyone interested in changes in subjective experience has given it a try already.
Yes, I've tried meditation but it doesn't lead to the marked increase in energy I'm looking for, just less emotional reactivity and a certain level of cognitive detachment that's useful in thinking within conversations. Both useful, but serves different purposes in my experience.
I've heard of Yerba mate but haven't actually tried it before. Curious to see how it compares to other teas, especially black/higher-caffeine ones.
I like mate and I mainly chose it over coffee because of the issues the OP mentioned. It feels like it works fairly well, I usually make a tea of 1-2 spoons and have it after waking up from night sleep and again after waking up from afternoon nap.
I have no idea how I'd do it but I think I'd like to try making mate gum at some point because apparently gum has benefits for focusing as well.
For what it's worth this perfectly mirrors my own experience. My performance increases with caffeine, I've not yet found a ceiling to the dose - performance relationship but I starting getting anxious and sleepless if I take it too consistently.
My body doesn't do anything on caffeine -- I took 600mg in pill form and almost fell asleep a few hours later. I know this isn't just me either; some people just don't respond to it. Perhaps there's another end to the extreme that causes experiences like yours. My gut sense is definitely that what you described isn't the standard coffee experience.
My life took a turn in 2015. That was when I first started drinking coffee on a regular basis. Not Americanos or Lattes, but 'Kopi' brewed with higher-caffeine Robusta beans using a 'sock' immersed in water close to boiling temperature, extracting massive amounts of caffeine along with strong bitter notes that are neutralized by sweet, sugary condensed milk.
My rate of learning in sports shot up seemingly overnight. My field of vision and attention increased tremendously. I gained more stamina and muscle control. It wasn't just a subjective feeling. The results showed in the speed and accuracy of the pitches I threw. The first training session I had with coffee was so productive, I almost swore that I would never train without it again.
Seeing its benefits, I began to use it for my studies as well. The hit of dopamine I got from starting problem sets with a cup of kopi was nothing like I'd experienced before. While in the past I had to start slow and enter 'flow' gradually, this was a shortcut to the flow state. Sure, learning didn't happen instantly, and I still had to struggle, but I enjoyed the process. I felt unnaturally productive; my absorption of information shot up, and I could go at it for progressively longer hours. This was my NZT pill.
Gradually, the more I was able to accomplish, the greater the aims I set for myself. I couldn't believe it. As a result of drinking coffee, my 'ambition' increased because my self-judgement on learning ability and productivity increased. I had other strong reasons for working hard, but the biochemical changes that occurred in my brain and body after ingesting coffee brought an emotional multiplier to those reasons.
However, this is where complications came in. I slowly began to build up a heavy tolerance to it. When before it raised my energy and loosened my social inhibitions, I started "zoning out" with friends, craving for that focused mode where I could work on my studies. Initially, I attributed the gradual shelling up to the long hours of study and mental stress caused by the major exam at the end of the year. Only after abstaining from coffee for a period of time, did I think of linking that never-before-experienced social discomfort to coffee.
On hindsight, I realised that I might have been experiencing a mild but non-negligible form of anxiety that came from work-related stress, compounded by the spikes in cortisol and adrenaline my body was producing in reaction to coffee. The initial spike of motivation in getting things done was offset by the dreaded caffeine crash. The sharp contrast of focus and pleasure in pre-coffee and post-coffee states was what got me into daily consumption, but in the long run, consuming coffee at that rate was a net detriment.
This leads to a question I have been trying to solve, and would like to pose to the community: if a common beverage like coffee can cause such a marked increase in life satisfaction and work/study output, what other foods or practices can lead to the same changes in subjective state, and is sustainable on a time scale of decades?