Hey, first post.
I have social anxiety disorder, and before I got on medication, I used to have panic attacks when physical altercations started, because of all of the attention that it drew. I had panic attacks at other times, but that's irrelevant. Try Zoloft.
I have to question Victor Smith's connection between training at a dojo and being good at fighting. If you want to be good at fighting, you have to be comfortable with hurting, maiming, and possibly killing other living, breathing human beings. Not only do you need to be comfortable with it, but in a fight, it must immediately become your primary focus. Many karate students can get taken out by preemptive headlocks and the like because they do not *train* in realistic situations.
Next time somebody pushes you, just duck into a protective stance and kick him the shins, hard, again and again. Unless you routinely square off against Muay-Thay kickboxers and such, it should put him down. Then run.