Usage of 去 and 來 : is "context switching" allowed?

Question

I am often confused with the use of 來 and 去. I made a picture synthesizing my current understanding of 來 and 去:

去 is used in the context when someone is going far away from A.  
來 is used in the context when someone is approaching a place (B in the picture).

But how to use this when A is talking to B about this? He could say « 我明天去B找妳 ».

But is context switching allowed? Is A allowed to pretend the reference is B and therefore say « 我明天來 »?

Whenever I've tried this, the other one (native Chinese speaker) told me that it was not correct and that I should rather use 去.

So, in my two sentences above (in gray), who is this "someone"? Is it always the one who is talking? Or is « context switching » allowed?

Answer 1

If you are at my place and you want to come to my place tomorrow, you should say 我明天来。Because you are right there where you are talking about. If we are at your place and you want to come to my place, you say 我明天去。Because it's where you are not currently at. Use 来 and 去 based on where you are and where you refer to.

Answer 2

It might depends on the recipient's location (B in this case) the moment the conversation takes place. For example, consider the below 2 sentences

1) I am coming to your place tomorrow (我明天來妳處)
2) I am going to your place tomorrow (我明天去妳處)

Both have the same meaning, yet if the recipient is currently away from his/her place, you would use 2) and otherwise 1) is used.