Does anyone know of any open source code for segmenting Pinyin syllables that has diacritic tone marks?
Example:
yìrúfǎnzhǎng -> yì rú fǎn zhǎng
C#, Python, or Lex is preferred but any language will do.
I've searched the internet but I'm coming up dry.
Edit: This is the algorithm I'm thinking of implementing myself:
Sort the standard list of all 414 syllables * 5 (tones) distinct syllable+tone combinations by length. Starting at the beginning of the text, look for the longest pinyin syllable that matches. Once a match is found move to the end of the syllable and repeat. Of course, there is the problem with syllables like xian matching xi'an. This happens because xian is longer than xi. For short passages of text and given the expected number of syllables I think I can figure this out by backtracking.