As i agree with the above, you did do the right thing. But if this is a constant issue where these people are harrasing you, why not put charges against them? Trying to physically fight them will be a huge mistake because they will pommel you half to death, but getting restraining orders against these people is not.
Seriously think about it, since you do have rights my friend
Be the change you wish to see in the world