If a human sat next to me I'd remove it. Usually I try and get seated first as I always get the window seat and I like to be settled. Put it up; fattie? Look at them, the divider, them again then sunglasses on and lean back to doze. If they are human sized? "Sorry! No need for that" with a smile. Then lean back happy you beat the odds on this day! (If flying in the US of course...)