It doesn't particularly matter if a specific kernel module has a big community behind it. 3-4 motivated people would be plenty to maintain a module. The major part of a GPU driver is in userspace, not the kernel.
The kernel community have slowly started to embrace rust in recent years. Maybe the community is not so small as you think, or at least won't be in the near future.