It did happen to me a few times in Japan at cafes and stuff, like people sitting with me when it was crowded. The first time I tried talking to the guy, but I either broke some social etiquette of not talking or maybe he wasn't expecting the gaikokujin to know some Japanese? Because he was visibly not into it