I think you missed the point; he didn't turn down the invites because he's gay, they called him gay because he turned down the invites...

I think you missed the obvious joke

If you've been to Thailand, there's no reason for it to be said jokingly, at all.

Yeah, I didn't mean it disparagingly or jokingly. Some places in Thailand quite openly cater to all proclivities. (Once when I was backpacking there, some guy came out of a side alley and approached me saying "young girls, beautiful girls!" and when I said no thank you, immediately switched to "young boys, beautiful boys?")