A relatively quick way to test that theory might be to pull down a WSL2 image of some sort and run the same thing in there.
Speaking as somebody who mostly codes for *n?x but has some windows users of their libraries, I've run face first into "everything I know about build optimisation is inapplicable on windows" more than once.
('everything' is admittedly slightly hyperbolic but I'm sufficiently bad at windows tooling that it always feels that way)
Sounds like something is messed up with your zig cache. I have a medium-size project (30 files, 10s of thousands of lines of code) that takes 0.07 seconds to `zig build` if nothing has changed. This is on a Mac.
Maybe this is a Windows issue? If I run "zig build" and then immediately run "zig build" again it still takes 3 seconds.