I think that verbs like 到, 去, and 来 are not durational verbs like their usual English counterparts "arrive," "go," and "come." For instance, they cannot be used with 在 to mean "is arriving," "is going," and "is coming." The Mandarin expressions do not describe the passage of time and only describe a punctual event involving a destination. As a result, 到 could not correspond with "as my wife and I were going/on the way to..." without other words to change its use in the sentence.
These Mandarin verbs can occasionally correspond with English translations using such progressive constructions. For example, to describe an imminent future event, you could say 警察来了 as an equivalent of "The police are coming!," but the underlying structure of the two expressions is quite different in the two languages and the meaning of the Mandarin expression in this case is heavily dependent on the contextual interpretation of 了. You might even say 他到一家火锅店去 as an equivalent of "he is going to a hotpot restaurant"; however, the Mandarin, unlike the English, cannot describe the action in progress and cannot mean "he is on the way to a hotpot restaurant." The Mandarin has more the meaning of "What he is doing (today) is going to a hotpot restaurant" or "He "generally" goes to a hotpot restaurant," depending on context.
I think the easiest way to understand the original Mandarin sentence is:
When my wife and I got to a self-service hotpot restaurant to eat a
few days ago...
The use of 到 with 时 forces the interpretation that the subsequent clauses happened after their arrival.