The sentences are from New Practical Chinese Reader 2. In the first case, the original snippet is:
(Note it's 还 = "return", not 换 = "exchange".)
You can see from the image that he hasn't yet returned the book, so it's definitely not a completion 了. It looks like a change of state 了: originally he said 我们就去借书 = "we're are going to borrow books", but realized he has to return his book first.
(On the other hand, it could just be a 语气词 (modal particle), because 把上次借的书还 without the 了 sounds unnatural or even wrong.)
Again, we can see from the image in the textbook that they haven't gone upstairs yet, so it cannot be a completion 了. I think it's a change of state 了; basically they were doing something (checking out the Beijing library), and now he's saying they need to do something else (go upstairs and get a library card).
I suggest avoiding relating 了 and "tense" (the keyword here is "aspect", not "tense").
I'm leaving. [present tense]
Wait until I get home, and then we can speak. [future tense]
I didn't go to the park yesterday. [past tense without 了]
In regards to...
And if 了 acts as the completion marker, can it be used with present and future tenses?
I think it would be fairly rare, if not impossible, to have something considered "completed" when we're talking about the present or future. It sounds like it'd just be a grammar error (outside of science fiction with time travel).
Also completion 了 is not used with all verbs. My teacher told me a good example of her student getting it wrong once: if I remember correctly her grandfather had passed away, and her student asked her:
What did he like to do?
The other thing that students don't realize (at least, it took me a long time to realize) is that, a lot of the time 了 is not used:
Yesterday, I saw a deer.
Yesterday, I studied hard.
This is unlike e.g. past tense, where it's far less optional.