The correct sentence is "我想退了这件新买的毛衣。" But why? Please do not use grammar jargon because I seldom understand them. I have been teaching Mandarin with grammar included but never mention and grammar term in my teaching and even the word "grammar" itself, simply because I am a victim of (English) grammar: hours spent just to memorize those stupid names, mind you, both in English and Chinese and their function/usage. Instead, I use "logic" to teach grammar because Chinese is one of the most logical and cultural language in the world.
Semantically, 我想 itself connotes a completion already because when you say it you already think about it. So we don't need 了 after 想 for completion. That's why the sentence in title is not correct.
退了 denotes I want the action 退 get done. "Done" is for completion, so we need a 了 after 退.
P. S. I somehow agree with you Chinese language is easier to learn if you focus more on logic, convention and culture, especially for second language learners. Chinese grammar is the hardest part of the language. Only few people truly master Chinese grammar. Second language learners can easily create a wrong/stilted sentence based on the grammar rules they learned.
Thought is not instantaneous.
Tom thought for a long time.
Wendy thought for a while.
Thought can take a while.
了 may indicate repetitive action
He thinks a lot.
Thoughts, ideas, have almost no existence, but they are the most powerful things.
In your case: 想 = 想要：want
want is a desire, but the thing you want has not happened or come to you, or you would not need to "want" it.
想 = 想要 is on-going, in no sense "complete" and not repetitive. 了 has a meaning "finished".
I want to return the sweater that I just bought.
(You have not yet returned it, but you "want" to return it.)
I want to return the pullover that I just bought.
(You have not yet returned the pullover.)
I wanted to return the pullover I just bought, but now I want to keep it.