Yes, confusion is possible. The long forms 第X声 "the Xth tone" and X个声调 "X tones" are unambiguous, but X声 can mean either "X tones" or "the Xth tone".
In Chinese linguistics, when discussing 方言 topolects, X声 usually means "X tones" as there is a standard set of terms to refer unambiguously to each individual tone (阴平 etc). You can number the tones but nobody does it as there are multiple conventions which means people just get confused. You more commonly see numbering in romanization schemes, but these all have their own well-understood definitions, are only used for speakers of that topolect anyway, and (in my experience) only as tone marker substitutes in the romanizations and never read as numbers.
These don't apply to the average Standard Mandarin speaker. To them there's just four tones and no possibility of anything else. So, people really only need to talk about which of the four tones it is. And technical terminology is, well, technical, so people are just taught to use the simple system of numbering tones. To ordinary people, X声 means "tone X" unless the context indicates otherwise (though I think laypeople are much more likely to say X个声调 for "X tones").