Well, this is what Japanese speakers do when they look at a Chinese text – they have some understanding of it since they recognize the characters.
One fundamental problem, though, is that in modern Chinese, the majority of words are made up of two characters. There are two types of dictionary for Chinese, one that gives character meanings (字典) and one that gives word meanings (詞/辭典). For reading, you need the latter type. Knowing the meaning of the characters is not enough to comprehend texts.
One reason for this is that characters have multiple meanings. For example, the character 節 can mean something like (A) ‘joint, section’, or (B) ‘season, holiday’, or as a verb (C) ‘to restrain, control’. There are numerous two-syllable words containing the character, and if you’re not familiar with the word, you can’t tell at a glance which meaning applies. The character can also appear in either position.
環節 ‘link’ (meaning A)
節奏 ‘rhythm’ A
情節 ‘plot’ A
情人節 ‘Valentine’s day’ (meaning B)
節日 ‘festival, holiday’ B
過節 ‘to celebrate a holiday’ B
節省 ‘to save, economize’ (meaning C)
節儉 ‘thrifty’ C
節育 ‘birth control’ C
(Note the different parts of speech of the last three – having 節 in first position tells you nothing in this respect.)
This is why most people who learn Chinese as a second language spend quite a while building up their active (spoken) knowledge and getting the most common words down before they tackle texts. Looking at real Chinese texts, it requires substantial background knowledge to even sort the characters into words. Trying to learn to read first would quickly become overwhelming, unless you have a very unusual mind!