This is an interesting question. As other commenters have noted, the answer may depend on convention (e.g., "回家") or grammar (possession vs. adverbial modification). It also may depend on the rules for Chinese prosody, which is what I will focus on here.
Tang Dynasty poetry was based on disyllabic (grouped into two syllables), trochaic (grouped into pairs having the form STRESSED unstressed, or, as I'll write it below, Xx) rhythms. Modern Chinese speech follows this basic pattern. Generally, Chinese utterances start with a stressed syllable (though a single unstressed introductory syllable may be used) and follow an alternating stressed/unstressed pattern. (One or two unstressed syllables may be used between stressed syllables, but having two adjacent stressed syllables is rare.)
Yip Po-Ching and Don Rimmington's Chinese: A Comprehensive Grammar gives the following example:
I'd like to buy another glass of wine.
This can be analyzed as 我想 // 再买 // 一杯 // 酒, with stress pattern Xx // Xx // Xx // X.
If the speaker wanted to add the word 去, the rhythm is disturbed:
The most natural option is to parse this as 我想 // 再去 // 买 // 一杯 // 酒, but this is rhythmically unsatisfactory, since it leads to a stress pattern of Xx // Xx // X // Xx // X. Both 再 and 买 should be stressed if the sentence is to maintain its emphasis on "buy" and "another", but this is not possible within the trochaic rhythm.
The solution is to delete the syllable 一 and say,
This pairs up nicely into stressed/unstressed trochees: 我想 // 再去 // 买杯 // 酒; Xx // Xx // Xx // X.
Chinese speakers naturally add and delete words like 的 or 一 to fit the basic stressed/unstressed rhythm. The subject is complicated, and I'm not sure how well its completely understood. The difficulty for language speakers who aren't attuned to rhythmic variations is that it seems like the deletions are random, when in fact they're often manifestations of underlying rhythmical rules.