They can't come in unless you give permission.  Just make sure that everyone in your family understands that under no circumstances should anyone be let in the door.  Do not get macho and show them a weapon, or they will return in force.