In my opinion, when the times comes and you're in the ring; it's just you that's inside getting beaten.
I've never been a 'loyalist' to any dojo/dojang/gym because if I feel the time is right that I have to move on to persue more opportunities, then that's what I'll do.
In your case, I'm not sure if you're training for fights, but if the schedule suits you, you should stick to one that you feel most comfortable with. Sure, you'll lose one out but heck, when the time comes it's just you. The team can only help you with the training (which is a large part of fighting), so get a team that's more forgiving, more supportive and most of all you can actually have a decent talk with.
Or you can go back to your main gym, talk to the management if it's possible to arrange a lunch-time training because of their schedule clashing with yours. If no, then move on.
Food for thoughts...
I got two fists.. Don't make me use my head as well!