I know this, and I hear you. But it also causes a headache for genuine freelancers who have multiple clients and have genuine overheads and expenses.
They’re exactly the people who should not fall foul of IR35, in a fair system, imho.
Multiple (unconnected and not contrived) clients, a clincher for me.
Unless its the equivalent of two or three part jobs, where they’re under the control of the different employers on their days working for them.