Would start with not inviting groups of people to go somewhere...
best to behave like a guest and quietly follow rather than turn up here and lead.
Sorry, I know that wasn't your intention.. but your western style stands out like dog's balls to someone living in Japan.
Just be as quiet as possible and follow what everyone else is doing.
Wait until you are invited.
Follow the conversation they start.. listening more than talking, of course.
The average newbie who comes here is usually too full on for the average Japanese person.
You should also watch all of these: