So, I can't help thinking that there must be something very wrong with me.

This line alone is a good enough reason to leave. Martial Arts is supposed to improve one's self confidence, not diminish it. It shouldn't matter that he gets on with other students - bullies have friends too.