Almost all the other characters with the 雨 component have relation to rain, clouds, and weather. However, this character 霉 seems to be the exception. It means mold, but the rain semantic seems very confusing. What does 霉 have to do with rain? And where did the term 倒霉 (fallen + mold) come from which came to mean "unlucky"?
「黴」 (Zhengzhang OC: /*mrɯl/), blackening from being afflicted by prolonged rain. From semantic 「黑」 (black) and reduced phonetic 「微」 (/*mɯl/).
The character 「霉」 (mould, mildew) did not exist originally, with the word that it represents being originally written as 「黴」. From character definitions, 「黴」 originally referred to the dark mould spots on garments from prolonged exposure to moisture, such as when rain hits and causes an environment to be damp. This is the direct correspondence between mould and rain.
The correspondence between 「霉」 and unlucky is less certain, with the word being written variously throughout history as also 「倒楣」 and 「倒煤」. You can find folk etymology explanations, but the choice of the second character for this word is most likely just a phonetic loan.
You need to do some linear thinking to understand the extended meaning of a word
下雨天时 --> 空气潮湿 --> 东西发霉
When it rains --> the air turn humid --> things get moldy
According to this story, 倒霉 was originally written as 倒楣
But we can still interpret 倒霉 as 倒运， 霉运 = bad luck / unlucky
霉运 = 烂运， 坏运 -- things get moldy (霉) will turn rotten /bad (烂/ 坏)