No -the good news is that you Leavers will benefit from immigration, just like the rest of us! And remember, the correct figure for net EU migration is 185k, not 330k. Just another little "error" which the Brexit camp like to make. Non-EU migration to the UK has ALWAYS been higher than EU migration.
So you think that 330,000 NET, the population size of Coventry and bigger than Brighton and Hove year on year, whether EU or Non EU is ok then.