It doesn't have to be an all or nothing proposition with C# or C++ anymore if Windows is your target environment. You can write the portions you want to write in C# and mix it in with C++/CLI. Yeah, I know, a lot of people think C++/CLI is ugly, but in the .NET world, it is a very clean glue language if you can get past the strangeness of having two different type systems (native and managed.)

You can essentially choose and manage how you want to deal with memory by virtue of how you choose between native and managed types, as well as control the behavior of the garbage collector itself.