Hi!
Why do you think the problem you are talking about is specific to you and not for everyone?
Without more details, the described problem looks like a common problem for everyone: most people struggle to formalize their thoughts (and programming is a formalization). No one can translate their knowledge into code without mistakes. People make mistakes; it is okay.
Are you comparing yourself to someone else? Or to some "ideal" image of "how you should work"?
Maybe you could provide more concrete examples of your problem, like "I need to do X, but I struggle here and here and make mistakes here and here."
>I'll have an example that I personally experienced recently, omitting some details related to my PII.
Based on this example, could you compare your expected outcome with the real outcome? In numbers. For example, "I expected to finish this refactoring in two weeks, but it took two months" or "I expected to finish this refactoring in two weeks, but my colleague completed the same task in two days." Something like that.
I have more than 15 years of experience in software development, not precisely in data processing but in gamedev and backend (and sometimes in data processing), which is pretty close.
From my experience, most people are struggling with the task you described. Even senior developers with 5-10 years of experience. I saw only a few people, who could do such refactoring naturally and from the first try.
I may suggest a few things:
- Try to reevaluate your expectations. Maybe you are too hard on yourself. You may overestimate the required result and try to do more than you need.
- Research a few methodologies of development. I may suggest Test Driven Development and Pair Programming. The first may help you control changes easier, and the second may help you learn how other people solve similar tasks.
- Try to pay more attention to code decomposition and structuring when you work on "pipeline prototype". Aka, add some "fake" functions/objects that will be required later, but for now, do nothing or contain trivial logic. Sometimes, it is easier to see what you need to do when you are fully in the task's context rather than when you are refactoring your code.
- Do not try to implement all features at the same time. Choose one, implement it, cover it with some tests or types or both, and then move to another.