Immigration is sometimes portrayed as solely a matter for the immigrant and his employer, or his landlord. Why should anyone else have the right to stop the immigrant moving here to get a better job, or to stop the employer hiring the cheapest labour he can find, regardless of nationality? Why should anyone else have a right to stop a landowner selling or renting his property to a foreigner? In a previous post, I explained that when the immigrant has negative externalities, your neighbours do have a legitimate interest.
The libertarian viewpoint under which immigration is solely a matter for immigrant and employer, or immigrant and landlord, only recognises freehold property, and property derived from it (e.g. leasehold). If you want to sell your land to a foreigner, no one else should be able to stop you, or so the argument goes. But this argument assumes its conclusion. Why shouldn't your neighbours be able to regulate what you do with your property? Why shouldn't there be collective property rights, where your neighbours can deny you the ability to sell your property to a foreigner?
Some immigration has nothing to do with business and employment. There are some immigrants who have no intention of working at all: beggars, con-artists, etc.
Of course it is not just immigrants who have externalities. All people do. Immigration control prevents bad people from getting in, but what about the bad people who are already here?
People are negative externalities by default unless they can justify their existence to me. They take up space, cause pollution, and bid up the prices of scarce commodities like prime locations, oil and other commodities. This is called the pecuniary externality.
Some people are positive externalities: they are so productive and do not capture all the benefits of their productivity. Inventors and innovators seldom capture all the wealth they create. Garett Jones' book "Hive Mind" explains how national average IQ matters more than your individual IQ; your having a high IQ benefits your neighbours more than it benefits you.
Beggars and other unproductive people are attracted to cities for the same reason that unproductive people are attracted to nice countries. It is better to be a homeless beggar in Britain than in, say, Libya, particularly if you are a woman. Begging is an ecological niche that doesn't work outside cities; there is not enough footfall.
The same applies to crime. Criminals tend to move to cities because there is more opportunity for crime there. There is more crime and criminals in the cities. Criminals don't tend to travel much, hence "white flight" to the suburbs where it is safer.