Everything can be put into rhymes to help people memorize. Maybe not poems or songs, just 顺口溜 (doggerel or jingle).
There is a meme phrase "满嘴顺口溜,你想考研啊". People preparing for postgraduate entrance exams need to memorize a lot of stuff so they make a lot of 顺口溜.
There could also be several variations for the same knowledge. When I was in middle school, we learned the dynasties with this:
夏商与西周,东周分两段。
春秋和战国,一统秦两汉。
三分魏蜀吴,二晋前后延。
南北朝并立,隋唐五代传。
宋元明清后,王朝自此完。
For the heavenly stems and the earthly branches, I don't know any 顺口溜. Maybe it's too easy. I mean, there are only 10 characters for the heavenly stems and 12 characters for the earthly branches. For the earthly branches, it's also common to combine them with the 12 animal symbols, which may be easier to remember, e.g. 子鼠丑牛,寅虎卯兔,辰龙巳蛇,午马未羊,申猴酉鸡,戌狗亥猪.
For provinces:
两湖两广两河山,
五江(疆)二宁青陕甘。
云贵四西北上天,
重蒙台海福吉安。
外加澳门和香港。
A very famous example for knowledge that is not a list: 奇变偶不变,符号看象限. It's about something in maths. It's funny that many people have forgotten about the knowledge but still remember this sentence.