Schuessler's reconstruction of the Old Chinese reading of 二 is *ńźi. It seems there is a correspondence between OC syllables beginning in *ń or *ńź and Mandarin syllables romanised as er.
However, I've always wondered at what the intervening steps in the sound change were. Cantonese has ji6, which maintains the nucleus. Similarly, the kan-on (漢音) borrowing of 二 into Japanese has the reading ni.
How did the rhotic syllabic vowel develop in Mandarin from *ńźi?
BTW, I have Schuessler's book, and the transcription shown in parentheses after the character indicates the Middle Chinese rather than Old Chinese pronunciation, so ńźi refers to MC rather than OC pronunciation.
Back to your question, it's important to note that initial ńź- is being used as a transcription rather than an indication of the actual pronunciation. According to A Handbook of Old Chinese Phonology, p. 55, Karlgren chose ńź- to transcribe this sound in order to reflect the fact that it evolved into a voiced fricative in descendants such as Mandarin (pinyin r- is often analyzed as IPA /ʐ/, a voiced retroflex sibilant); however, the book notes that "for Early Middle Chinese, however, it is widely agreed that it was simply a palatal nasal (IPA: /ɲ/)."
As you noted, this Early Middle Chinese /ɲ/ became /j/ in Cantonese, and /n/ in Japanese. Such phonological changes are not too hard to imagine occurring, though EMC /ɲ/ changing to Mandarin's retroflex /ʐ/ may be a little harder to imagine. However, it is interesting to note that in Late Middle Chinese, all palatal sibilants merged into the retroflex sibilants (e.g., /tɕ/ -> /ʈʂ/, /dʑ/ -> /ɖʐ/, etc.). At this time the palatal /ɲ/, while not a sibilant, also got carried along with this change and became retroflex in LMC. After that, metathesis occurred which changed /ʐɨ/ to /ɨʐ/ (see 《北京音系解析》, p. 24), which is realized as [əɻ] (pinyin: er) in modern Mandarin.
As an interesting aside, phonological changes often have exceptions, so a few words continued to retain the initial nasal in their modern descendants, the best example being 爾/尔 (MC: ńźje, Mand.: ěr, Cant.: ji5), whose colloquial pronunciation retained the nasal initial and became modern 你 (Mand.: nǐ, Cant.: nei5).
Claw wrote:
After that, metathesis occurred which changed /ʐɨ/ to /ɨʐ/
I'd like to add that Karlgren envisioned the crucial step as vocalic epenthesis (insertion of a vowel) rather than metathesis (inversion of two segments). He envisioned a stage in which these syllables were syllabic /ńź/ through loss of the final vowel, after which a schwa was inserted before the consonant.
