You need physics in order to make movement server authoritative. Otherwise player will be able to fly around and ignore collusions. You might need animation state in server if you are making a fps shooter and want to calculate raycasts properly.