It sounds like you knew these people, maybe he was familiar with your martial arts training. Most guys that beat up on women don't like to fight men.

It is interesting, you spend so much time training and just your stance and look make people turn an walk away from a fight. I have done that numerous times, but i am a big guy so taking a stance or giving a look usually stops most fights. When they know you are prepared, they are not willing to risk it. So practice and train hard, knowing your immediate reaction, preparation for battle can and does change an adversaries dispostion. I have had very few fights less than five in my life all defensive. How many fights do you think you might not have even had because the guys looking for one can just see it in your stance or the way you handle yourself. You dont realize it but the training you receive is perceived in your body language. Be happy you have put the time and effort in.