You got the general meaning right, but because some details are left out, it is harder to understand the meaning of “只” here. And the context does affect the interpretation.
Basically, "只" here means only, which could be interpreted as the only thing you do since the first part says there isn't even a word/news from you after you left.
To better illustrate it:
Why (I) never heard from you after you left?
(You) only left memories accumulating in my heart.
OR: The only (thing you left) are memories that accumulate at the bottom of my heart.
Here "思念"(the feeling of missing someone) is translated as "memory" for connivence although they are not exactly the same.
So if I have to put the two parts in one sentence, it would be something like:
"Why did you leave without a word except letting memories accumulate in my heart."