ESC is the key that (roughly) takes you back to normal mode from insert mode, visual mode, etc. If you aren't using it, then you're either using one of the other ways to get back to normal mode (like CTRL-[), or you are using Vi in a very non-standard way.

See, for example: https://en.wikipedia.org/wiki/Vi#Interface