But it wasn't an attempt to make a foul either.
That's irrelevant. If you make contact and you're not trying to play the ball then it's a foul and a red card. If he'd tried to play the ball and fouled the player then it's a yellow card. The law is quite clear on this