I dont look for fights obviously, but theres been cases where iv backed down when i really didnt want too because i no im not tht big and would most likely get messed up.
So yes, basically i want to assert myself better and not be so affraid to fight.
Not looking for fights and controlling your ego when things get heated so you don't get in to a fight is exactly the sort of thing you should be doing. That sort of stuff IS self defence. People can avoid trouble and stay out of fights there entire lives without ever taking martial arts classes.
That said, if you want to control your stress and fear, will say this one more time: SYSTEMA
Systema is a Russian system that originated from the Soviet Military. It involves the concept of "total relaxation". Aside from fighting skills, they teach exercises to help a person relax mentally and physically even in stressful situations. As I said, you have one of the best instructors in the world in Toronto. Would really, really encourage you to try at least one class.