This is probably a better question for linguistics stack exchange; however, I will say that you are not hearing things incorrectly. My background is in linguistics, though not specifically in phonetics and phonology, so don't consider this an expert opinion. However, given some searching, I've found a helpful Wikipedia (Glides section) note on this:
The glides may occur in initial position in a syllable. This occurs with [ɥ] in the syllables written yu, yuan, yue, and yun in pinyin; with [j] in other syllables written with initial y in pinyin (ya, yi, etc.); and with [w] in syllables written with initial w in pinyin (wa, wu, etc.). When a glide is followed by the vowel of which that glide is considered an allophone, the glide may be regarded as epenthetic (automatically inserted), and not as a separate realization of the phoneme. Hence the syllable yi, pronounced [ji], may be analyzed as consisting of the single phoneme /i/, and similarly yin may be analyzed as /in/, yu as /y/, and wu as /u/.1:274ff It is also possible to hear both from the same speaker, even in the same conversation.1:274ff For example, one may hear the number "one" 一; yī as either [jí] or [í].
What this seems to be saying is that the glides [j] and [w] can be inserted in front of certain vowels of which the glide is considered an allophone and that the insertion of such glides depends on the speaker. From my own experience, this very much seems to be the case. I often speak with Taiwanese/Southern Chinese speakers and will almost always hear [u] to represent the pinyin "wu" and [i] to represent the pinyin "yi". For what it's worth, this is also considered "standard" in terms of the sources (1,2) I've found.
I am avoiding / / notation since usually this is used to denote phonemes (something used to distinguish minimal pairs), whereas [ ] is used to denote surface realizations of the phoneme (allophones). By the way, if someone more familiar with phonetics/phonology finds something wrong here, please do correct me.
To sum up, you are not wrong in hearing it this way. Just be mindful that you may hear some variations. If you want to be safe and certain that you will be understood, then pronounce them as [u] and [i]. Though, given some more listening practice, you may start to mimic some other accents. Try out this website to test out words like "五" and ”一“. You can filter by region to try to hear differences. Personally, I hear the glides inserted more often when using the CN filter, and find them less so when using the TW filter.
PS: You might also find it interesting to look at "Bopomofo" or "Zhuyin", the phonetic symbols used in Taiwan as opposed to Pinyin. In the case of "一" and "五”, both use symbols that correspond simply to [i] and [u] respectively.
EDIT: To answer your question "Are there (other) contexts where the word initial glide does not occur?". To my knowledge, this is the only context where [w] and [j] are omitted in pronunciation but still have "w" and "y" written explicitly in Pinyin. Someone can correct me if I'm mistaken here.