You're getting to the point where you really need to use some number theory and define "number" more precisely. I would imagine you could construct a definition of number based on multiplication whereby 0 would no longer be a number... but only according to your new definition of number. There's nothing wrong with having a new definition of number, but there is something wrong with writing the statement in English "0 cannot be a number" without providing your definition of number. By the standard one that English usually means when the word "number" is used without mathematical qualification, yes it darned well is, just as infinity isn't. And, correspondingly, you can construct numerous number systems where there are various ordinals that have characteristics inline with what we expect from "infinity", but those aren't the usual infinity anymore, either.
When somebody asks whether infinity "is a number", I don't think it's really helpful to start dragging in constructs from number theory and defining new definitions of number for the apparently sole purpose of dazzling the poor question asker. Either just answer the question: "Is the simple, traditional infinity I learned about in high school an instance of the simple, traditional number I learned about in high school?" ("No."), or be more explicit about the fact that you're not talking about either the "infinity" or the "number" the asker is asking about. (And when I phrase it that way, my objection should be clear: You're not answering the question that was asked, and you're not telling them that's what you are doing. That's not educational by any useful metric.)
Sure, neither "traditional number" nor "traditional infinity" is particularly well defined, with pathological behavior readily demonstrable to even a high-school educated person... but not in such a way that affects this particular question.
Take equation: ∞ + 1 = ∞
If ∞ is a number you can subtract it from both sides giving 1 = 0
So ∞ cannot be a number.
But then 0 too is not a number:
Take equation: 0 * 2 = 0
If 0 is a number you can divide it into both sides giving 2 = 1
So 0 cannot be a number.
0 is special. You cannot multiply and divide by it unconditionally.
∞ is special. You cannot add and subtract it unconditionally.