Nice idea, but surely many ex-players would have a lot of clubs they either like or dislike. And lots of players they're mates with or really hate from their own playing days. And sets of fans they have a problem with. So basically, whilst it's a good idea, how would you get impartiality?
Doesn't seem to bother cricket authorities. We frequently get old Sussex players at Hove (and other counties will welcome former players) and no-one seems to too concerned.
It could be that refs are professional enough to put any loyalties aside.