Well first off, try to avoid getting into street fights.
Particularly since it might not always be a one on one situation. If a guy comes to fight you and his friends with him see him losing badly, they might decide to help him. Then you'll have more to worry about then just the one opponent's abilities and rep.
Do you still practice now? You said that you studied an art for 10 years, but do you do any training for it now?
If you are training, I would suggest maybe participating in some tournaments. Whether you like them or not, you do get to face an unknown opponent. Doing that enough might help lower the level of the fear you feel. Also, if you are training, you can take some solace in knowing that you are at least doing everything you can to improve. If you happen to get beat, it's not because you didn't try.
If you aren't training of course, I would say try to get back into it. Especially if your this worried about being attacked. Maybe it can help you calm your mind a bit and get you more confident in your own abilities.