I doubt if many British citizens are doing the work that home country nationals do not want to do.
There may be legions of fruit pickers, and labourers here, but frankly, I doubt it.
That would indicate that EU residing UK nationals are not driving down wages for the poorest in their host country and of course the disparity in the economies and wages between the UK and some of those EU countires means there is no incentive for those to go there and work anyway.