角 (litterally a corner / a horn of an animal) = 毛 (litterally "a feather of a bird") = 1/10 the main unit.
Why use 角 and 毛 for, what we might call in American English, 10-cents?
角 came from 銀角, which was historically a currency that represented a fraction of the silver coin (銀元). 元 came from 圓, a description of the coin's circular shape. A theory for 角's use is that since the basic meaning of 角 is a horn; by extension it came to be used to describe "things that looks like horns". And from there, "corners" 角落, "angles" 角度, etc. Dividing a circle up like pizza slices creates a "角"; hence, the divisions of one 元 (圓) came to be called a 角.
As Carmelos said, 毛 came from a simplification of 毫 - it is the bottom radical of the latter. 毫 in turn comes from one of the traditional Chinese unit for small numbers: 分 厘 毫 絲, corresponding to the modern 10-3 (it is still used for millimetres, 毫米/公厘). A theory is that 毛钱 was a type of coinage in the Song dynasty that had very small values, as its name suggests. The term persisted in that meaning of a "small denomination coin", and thus became synonymous with the modern small 角 coins.
I don't know how good you are at Chinese, you can read this Chinese wikipedia page for more information: http://zh.wikipedia.org/wiki/角_(貨幣)
"毛" is simplified from "毫", which means little amount. In Chinese we have a saying as "一分一毫", which means very little amount. In most part of China, "毛" and "角" are interchangeable.