Colours are an abstract concept, and ancient Chinese largely used words for objects with the colour to describe the colour itself.
- 紅, 綠 originally referred to silks dyed in those colours, hence the use of the 糸 semantic component;
- 藍 originally referred to a plant used to make blue dye, hence the use of the 艹 semantic component;
- 黑 originally referred to a punishment called 黥面, in which a criminal had the name of their crime tattooed on their face in black, and was originally a pictogram depicting this;
- 青 originally referred to the colour of grass/plants, and the top was originally 屮 or 生, indicating this meaning;
- 赤 originally depicted a person (大) being scorched (火) maybe as a form of punishment, indicating the colour of burnt skin.
Sometimes characters were used as a phonetic loan for colours, and had no connection to the original meaning, e.g. 黃, 白. 粉紅 and 咖啡, as you've pointed out, are modern terms, but they reflect the same idea of using things that have the colour as a word for the colour itself.
Although it did appear with the meaning colour, the use of 色 was unnecessary in ancient texts. However, in a general trend of words evolving from single syllables to double syllables, 色 came to be used in conjunction with the above characters to form a combined meaning of X (colour).