It took a lot of time for most C++ compilers to get good at removing abstraction. It was so bad that Stepanov wrote the abstraction penality benchmark to name and shame compilers.

In particular vectors with class type iterators (as opposed to using raw pointers) paid an heavy penality. Is it possible that your vector used raw pointers as iterators?