This.You are right, but you're overemphasizing the importance of that association.
Some people ARE celebrating the festival of Christmas. Some simply the 'Christmas holiday'. They'll call it that because that's what it is called, not because it has any significance to them. The quote you used earlier about being 'Culturally Anglican'. Well, that. The christmas holiday is part of our 'Anglican culture'. Believing anything is no prerequisite to joining in the party.