Lexical tones are known to superimpose onto the prosody of the intonational phrase to which they belong. They are often described as superficial crenellations on the surface of sea waves. The prosodic 'sea waves' wax and wane according to communicative intention and it's usually the phrase in a sentence with maximum focal weight which waxes or wanes the most. That will usually include the predicate together with any modal particles.
For example, when the communicative intention is one of polite interrogation, as marked by 吗, the intonation of the sentence (or just the predicate in focus) will be higher-pitched and this will lift the lexical tones higher up, especially weak-tone syllables. On the other hand, the intonation of an affirmative sentence or phrase, one marked by 的 for example, will push the tonal contour chain into more of a downward curve.
So I agree with your suspicions: this is ultimately related to the semantics of those particles. I would argue, as I have done, that those tonal differences you seem to hear are not somehow inherent to the particles themselves but are a result of the intonation associated with their use.
You might still wonder if those particles have inherited those intonational patterns to the point where intonation is now partly incorporated into the tonal profile of the particles themselves, even in isolation. It sounds unlikely to me, but it should be easy to prove/disprove in a study if anyone feels inclined to pursue the hypothesis further.
