There was no equivalent of pinyin in ancient Chinese, because Chinese writing developed as a way of representing the spoken language—at first for special purposes (e.g., recording divination), and then later more generally. If there had been pinyin in, say, the 商 Shang dynasty, they might well have used that instead of Chinese characters. Why have the (as you say) more complicated Chinese characters at all, if you already have a perfectly good representation in the pinyin?
This sounds a little odd today, because we're used to the idea of Chinese characters, and pinyin is used as a way to indicate their pronunciation, but English gets by perfectly fine using only its equivalent of pinyin (normal English orthography). Chinese could have done so, if they had had such a system, but keep in mind that the alphabet is a rather deep innovation, and it is by no means obvious, and we should not be surprised that they did not have such a thing around the inception of the Chinese writing system.
Probably by the Three Dynasties period, and certainly no later than the 隋 Sui dynasty, when the 切韻 Qieyun (the oldest surviving example) was created, people used 反切 fanqie as a way to express character pronunciations. It should be emphasized that fanqie is not spelling as we normally think of it in alphabetic languages. In principle, it divides Chinese character pronunciations into two parts—an initial and a rhyme—so that the character 東 dōng would be split up into d- and -ōng.
But these two parts would not be represented using new symbols introduced only for phonetic purposes, which would have been the beginning of a true spelling system. Instead, two other existing characters would be selected, one that shared the initial, and one that shared the rhyme. In the Qieyun, these characters were 德 dé and 紅 hóng, and so 東 was understood to have a pronunciation that shared the d- with 德 and the -óng with 紅.
To be sure, 東 is first tone and 紅 is second tone, but at the time of the Qieyun (Middle Chinese), both characters were level tone, and what we call level (first) and rising (second) tone are the result of a register split that happened in all four Middle Chinese tones, after the time of the Qieyun. Mandarin preserves this split of the level tone, but only keeps one side of the rising tone in its third tone, one side of the departing tone in its fourth tone, and loses (in most areas) the entering tone altogether. Other dialects, such as Cantonese, preserve most of the results of the split.
Note that fanqie merely expresses the pronunciation of one character in terms of two others, so just as someone who is ignorant of English would find it difficult to learn it from a dictionary, someone who is ignorant of all characters cannot learn anything about the pronunciation of any character from fanqie "spellings". (Also, as a side note, the Qieyun was not intended as a general-use pronunciation reference, but was rather dedicated to Classical readings.) We might say that equivalence classes of initials and finals are generated by the fanqie spellings, but no explicit values are denoted. That requires a set of specialized phonetic symbols such as an alphabet.
As far as the characters are concerned, they were not always so complicated; in many cases, the so-called "simplified" characters are actually the original form, and what we call "traditional" characters were later elaborations to distinguish two characters with the same pronunciation but radically (pun only partially intended) different meanings.
This came about in part because of the use of 假借 jiǎjiè, or borrowing, in which an existing Chinese character with (typically) a concrete meaning, and therefore a straightforward pictorial or representational depiction, was borrowed for another word with the same pronunciation. One of the most (in)famous examples is 來 lái "come", whose character represents a wheat stalk, a meaning now captured by the character 麥 mài "wheat". The simpler character was borrowed for the more common meaning, which led some people to speculate that there was a form lái that meant "wheat"—but in fact, the situation is simpler than that: The original character for "come" was 麥—note the presence of the foot radical at the bottom—so the two characters merely "switched places".
This kind of extended usage is fine if you only want to write something down to remind you of what you yourself meant, but if you want to convey something to someone else, it behooves you to specify it more precisely, which I suspect gave rise to the use of 形聲字 xíngshēngzì, or phonetic-semantic compounds, which account for the vast majority of Chinese characters (around 90 percent, depending on whose count you trust*). The phonetic components of many of these characters were themselves compounds already, leading to a large nested structure and the array of complicated traditional characters you observe. (This complication, by the way, is why on the mainland, you'll typically see traditional characters referred to as 繁體字 fántǐzì, to allude to their complexity, whereas Taiwan often refers to them as 正體字 zhèngtǐzì, to allude to their "correctness".)
In short, Chinese writing evolved organically, as the result of countless scribes working independently, and even with the efforts of the Qin dynasty to keep things straight, lots of characters were standardized that today seem pointlessly complicated. Whether simplified Chinese measurably helps literacy is a different and difficult question to answer, because that too was done somewhat organically and not always consistently.
Hope this helps. Some of this is just general knowledge, but a couple of (English-language) reference sources that I've found useful are
Ramsey, S. Robert, The Languages of China.
Schuessler, Axel, ABC Etymological Dictionary of Old Chinese.
*Interestingly, the percentage varies from source to source, but there seems to be general agreement on how many characters are not phonetic-semantic compounds—about 1500—and so the variation in percentage arises almost wholly from how many characters are reckoned to be used in Chinese.
