Another way that might work to explain this is that the basic concept at work here is that the colors ending in -色 are the names of the colors as things of interest in themselves, whereas when we omit 色, we do so when we are using colors as attributes of another object of interest. We might call the latter adjectival, but also includes attributives.
For example, if someone asks, "What is your favorite color?" and I were to respond with "Red", but we're using Mandarin and not English as medium of communication, I would use "红色" as the response, not just "红". In this context, the object of interest is the color red itself. On the other hand, if I want to talk of "red apples", then I would say "红苹果" - here now red is serving to describe a quality of the objects of interest, which are apples.
This also explains @mootmoot's post, where the example
which means "I like the yellow ones" (those things to which can be attributed [的] the quality or characteristic of yellowness [黄]). The objects of interest are some unspecified-in-this-example-but-obvious-in-real-usage things for which color is an incidental or peripheral property. This is different from saying "I like yellow", where the focus is on the color yellow itself. This phrase would be, thus, said as