I've had that happen. Worst is when they are in a middle or window seat and you're forced to get up to allow them in. If they try lifting the arm rest, I push it back down before they sit because they are not invading my area.