They all mean "otherwise", and beyond that 要不 is more colloquially used I'm not sure how they differ in usage or meaning.
否则 is often time more formal and comes across as more strongly worded (directly admonishing the listener). Much like saying "you need to do X, otherwise/if not, Y will happen (as a consequence)."
要不 comes across as the softest (has the same as either/or, in the same manner as a suggestion), but it might the nuance of sounding as if one is being admonished, depending on the context. Much like saying "you (either) need to do X, or Y might happen."
However, note that 要不 is (more commonly) used to as a way to directly or indirectly ask the listener for consent (based on the context it's used). Seeing the 吧 typically will indicate that this is a suggestion.
不然 is often used when you want to assert a matter of fact (typically in a neutral manner), the manner of speaking is probably somewhere between 否则 and 要不 in terms of strength. Much like saying "you need to do X otherwise Y might happen."
要不然 is a about the same as 要不, but in a bit stronger tone than 要不. Much like saying "you need to do X， or otherwise/or else Y might happen." In my opinion, 要不然 can be used interchangeably with 要不, but only in when used in the context of "otherwise."
Let me clarify it a bit:
否则. 否=not(negative), 则=then. 否则=if not then.
要不然(要不 and 不然 are simplified forms of it). 要不=要是不=if not. 然=like this/that.
So there are almost no difference. You can switch one by another in modern Chinese. 要不然 is weaker than 否则 in general.