Yeah.  From a Non-Aggression Priniciple perspective, it imposes negative externalities.  From lower social trust to companies being miserable places to work because why change when you can hire scabs and don't have to compete for employees.  But the people who insisted on mass immigration don't have to pay for that.