In duolingo chinese course I came across this sentence. 他游了一个小时的泳. What is the grammar rule for separating 游 and 泳 ? And why "的" in "一个小时的" ?
It's because grammatically 游泳 is 离合词, which means it's separatable.
Baidu also gives other examples there, such as 睡觉, 吃饭，etc.
So, we can say 睡一个小时觉，吃个便饭，etc.
的 is 助词(auxiliary word) used after an attributive phrase. 一个小时的 is an attributive phrase modifying the noun 泳. 的 is omissible in your example.
PS. 泳 intrinsically is a verb, and here it's used as a noun.
You probably should not try to use any grammatical rules to understand the structure here. Since generally, We do not speak after trying to figure out what we say is a "离合词" or not and therefore, there are indeed lots of case that one says a sentence following the same pattern as the one you asked, however, which is not a "离合词." For example: "滑天下之大稽," here "滑稽" is by no means legitimate to separate. And here "稽" can definitely not be modified by an adjective like "天下之大(as huge as the whole world)" but since it was used by a famous Chinese proser Lu-xun, it was regarded as a classic way to express people's dissent for the things without sense. Why can it be separate? You may want to ask. The reason here is that it will neither lead to any ambiguity nor confuse any existing meaning and, what is the foremost, the usage here can express the writer's feeling in a perfectly exact way. To a certain extent, from the very beginning, Chinese is a language specifically designed for the extreme conciseness and transmitting both the feeling and the idea in an exact way simultaneously. To obtain it, we have to exploit and utilize each tiny structure in the sentence to serve for that somehow self-contradict purpose and, therefore, sometimes grammatical rule has no place to fit in. Similarly, Chinese people (both in ancient times and the present) are more pragmatic than logical and reasoning.
Then probably you want to ask, how could I know whether I can separate a word or not. My answer is that: First, as I said before, it should not confuse any existing meaning. Second, the first character must able to be used as a verb, even though it may not be used as a verb in the sentence(but it can be). And the third actually overrides everything, it can not be meaningless, and people can understand you. For example, "离合," you may want to separate it like "离天下之大合," to put it in a sentence, "离合也可以当做离合词? 这词可真是离了天下之大合呀！", and I will say yeah it is perfectly fun and fine. But you can not use it in a formal document except for some words people regard as a specific way to express something"游了十分钟的泳." for example)