Traditionally you'd use an optimizer, but OP isn't doing it traditionally, which is what makes it interesting. NeRFs work differently.