Yes there is. It's a moral wrong rather than a legal one but it is still a wrong .
Why pay tax than you need to? Tax avoidance is simply planning sensibly to minimise your tax bill. e.g. making a pension contribution to get tax relief, if a husband pays higher rate tax and the wife doesn't put assets in her name etc etc
As I said I think people are getting confused between tax avoidance and tax evasion. two very different things.