One other way to look at it: 我的 is not only "my," but can also be, "mine."
我吃饱了，你吃我的吧。
I'm full, so please eat mine.
Just like in English, in Chinese you could end the sentence with either "my <noun>" or "mine". When you hear this sentence and notice no noun or noun-phrase follows 的, this is perfectly OK, and you can translate the 我的 as "mine".
But unlike English, Chinese follows a pattern. You can put other things before the 的, too. So not only can you specify 你说的事情 "The issue you're talking about," you can instead say 你说的 "That which you're talking about," which is awkward in English but concise and natural in Chinese.