If my nephew(姪兒) and I dine with a relative, she is my mother's younger brother's daughter, what should my nephew call her in formal Chinese?
Let's call this nephew Peter, to simplify the explanation somewhat. Now Peter is your 姪兒, which means he is your brother's son. That will make your mother his paternal grandmother.
This female relative is his grandmother's niece (brother's daughter), which is also his father's cousin.
So your question is basically this:
What do you call your father's female cousin?
Your father's sisters are your 姑媽 (older) or 姑姐 (younger).
With cousins you add the word 表. If this female relative is older than Peter's father, she is 表姑媽; if younger, 表姑姐.
If you draw a little tree, it will become very clear. The relationship between Peter and this female relative does not need to go through you, and it is not as distant as it sounds.
Chinese kinship terms are not so complicated after all, I hope.
Edit: I feel compelled to add this to my answer even though it does not pertain to your question. I said add the word 表 to your sibling terms to turn them into cousins. Well, that's not entirely true all of the time. If, for example, this female relative were Peter's paternal grandfather's brother's daughter, then she would be a 堂姑媽 or 堂姑姐.
The difference between 堂 relative and 表 relative is that with 堂, the relationship is through an "all-male" cast. In our hypothetical case, this is Peter's father's father's brother's daughter. Not counting the two people whose relationship you're trying to figure out, every person that is involved is a male. Then this relative is a 堂 relative. As soon as there is at least one female involved, it becomes 表.
This begs the question, why the distinction. Well, the 堂 relatives have the same family name, and in a patriarchal society, it deserves a special term, I believe. Once a (married) woman is involved, the family name isn't the same anymore, assuming everybody takes their father's family name.
"If my nephew(姪兒) and I dine with a relative, she is my mother's younger brother's daughter, what should my nephew call her in formal Chinese?"
Your mother's brother's (舅舅的) daughter is your cousin - 表姐/妹. Now, if you are a sibling of your nephew's father, then he shall call her 表姑 (as for his father's cousin). Otherwise, 表姨, if you are a sister of his mother; or brother of his mother. But in the latter case, as noted by @monalisa, your nephew is your 外甥 instead of 侄子.