IN SHORT
Both words are indeed used. In everyday conversation they're usually not distinguished but have different usages.
If we go a little deeper, they also represent different colors: 橘色 has a more reddish hue than 橙色, which is more yellowish. This is consistent with our view of the color of oranges and mandarins. This can be corroborated with the choice of different words when describe the ripening of oranges and mandarin, i.e. 橙子黄了 vs. 橘子红了.
Standard Mandarin vs. Dialectal Usage
That being said, let's see what《现代汉语词典》, which is generally used as guideline for word usage in Standard Mandarin (as Standard Mandarin is not a natural dialect, I advocate the authority prescription even though it could be sometimes linguistically incorrect), says:
橙色：not found
橘色：not found
橙黄：像橙子那样黄里带红的颜色
橙红：像橙子那样红里带黄的颜色
桔黄：not found 桔 is the 俗字 for 橘
橘黄：比黄色略深像橘子皮的颜色
橘红：像红色橘子皮那样的颜色
This is consistent with disyllabification of modern Chinese. The latter four are preferred by native intuition. But as you know, Chinese has many mutually unintelligible dialects, which are historically merged with different local non-Chinese languages,and also inherit and develop different features of Old Chinese and Middle Chinese, dialectal usages could be very different. When people communicate in Standard Mandarin, or they think they are, dialectal habits are often carried over.
I'm native to Chengyu Dialect of Southwestern Mandarin. For my region (again, this is a general law and there could be exceptions even within Chengyu Dialects), 橙色、橘色、橙黄、橙红 (I incline to write 橘 as 桔 in Southwestern Mandarin but there's no orthography dictionary) are understandable but not used. Instead, 橘黄 is used for yellowish orange, while 橘红 is used for reddish orange. When I hear 橙色、橘色、橙黄、橙红, I would interpret 橙色=橙黄=橘黄 and 橘色=橙红=橘红. So in my area, we distinguish the color of 橙 and 橘 when used alone, but mix them if used to modify color.
I don't speak another dialect to be able to comment more on this.
Formal vs. Informal
If we search them in 现代汉语语料库检索 Modern Chinese Corpus, the following is found:
橙色：18 entries
橘色：not found
橙黄：10 entries
橙红：3 entries
桔黄：4 entries
橘黄：not found
橘红：not found
This corpus is not very comprehensive but the result is still indicative. It is also consistent with my intuition: 橙 are associated to a more formal usage than 橘.
Idiomatic Usage Across Dialects
橙 and 橘 also have different idiomatic usage.
For example, the color of the rainbow is 赤橙黄绿青蓝紫，橙 cannot be replaced by 橘. Even in my dialect which exclusively use 橘 for colors, 橙 is used here.
On the other hand, when we talk about the color of hair, either for the dye of human hair, or the hair of animals, say cats, 橘色 is used. This usage is so fixed that we call orange cats 橘猫, even for dialects which exclusively use 橙 for colors.
The color of Chinese paintings is also fixed usage. 橘红 and 桔黄 are used, not 橙.
