It's not a zero sum game. You're no worse off for someone going faster than you.
Apart from:
-more chance of you or a loved one being killed;
-more pollution;
-greater contribution to global warming;
-more enforcement of speeding due to idiots taking the piss; and
-more chance of getting held up in an accident.