的 generally expresses possession or modification and 得 is used after a verb or an adjective to express possibility or capability.
Here are some literal translations that should help to differentiate between the two examples that you listed:
他说的汉语很流利。
[他说的汉语][很流利]。
[The Chinese that he speaks][(is) fluent].
他说汉语说得很流利。
[他][说汉语][说得][很流利]。
[He][speak Chinese][speaks][fluently].
In general, that which comes before the 的 modifies what comes after it:
他穿的衣服很时髦。
[他穿的衣服][很时髦]。
[The clothes that he wears][(is) fashionable].
The 的 ends up forming a complex subject:
他说的汉语
The Chinese (that he speaks)
“他说” modifies "汉语".
Which Chinese? The Chinese that he speaks.
他穿的衣服
The clothes (that he wears)
"他穿" modifies "衣服".
Which clothes? The clothes that he wears.
红色的书
The (red) book
"红色" modifies "书".
Which book? The book that is red.
得 is generally used in the following pattern to describe the degree to which something is done: [Subject][Verb-Object][Verb 得][Adverb]
.
她看书看得很快。
[她][看书][看得][很快]。
[She][read books][reads][quickly].
[Subject][Verb-Object][Verb 得][Adverb]
With the dialogue that you provided, the "的" should actually be "得". It seems to just be a typo.
Lily 说:“我小吗?”
Lily: "Am I small?"
Monster 说:“小? 你? 你小的不能再小了。你非常小。”
Monster: "Small? You? You're so small that you can't get any smaller. You're very small."
With the third example, 得 should not be used. 的 is correct here, but it can be confusing because there is an implied/omitted noun. The thing being modified isn't said explicitly and all that is said is the modifier.
我不知道我做的对不对。
[我][不知道][我做的][对不对]。
[I][don't know][(that which) I did][is/is not right].
I don't know if what I did was right.
Here is another example dialogue:
A: 我要买书。
A: I want to buy books.
B: 新的还是二手的?
B: New (books) or used (books)?
A: 新的。
A: New (books).
Notice how "书 / books" is not repeated. Once the object is clearly in context, both speakers drop it for convenience and brevity. Natural human speech tends to be like this. It would seem quite odd to say everything explicitly since once something is in context, repeating it can seem unnecessarily redundant.