He wouldn't lose seats to them. The Tories will only field hard BREXIT candidates. But Farage will have a say in a non aggression pact despite winning no places in parliament.
This country is now run by a minority of right wing zealots.
One factor that no-one's mentioned yet is that what the 20-odd MPs who are kicked out of the party are going to do? Some of them are leaving parliament anyway but people like Rory Stewart and Sam Gyimah are quite young and won't be finished with politics. There's already talk of a new centre-right Conservative party being formed and there's every possibility that a few MPs find themselves standing against Tories ... which will be interesting.
The other question is what happens if Johnson loses his seat. I have a friend in Uxbridge and she tells me that the seat is a top target for Labour and they're very hopeful of winning it. BJ didn't win any friends locally with his cowardice over Heathrow and he's certainly not a shoo-in for it.