What would you conclude to be the single most important issue/hurdle/constraint about Angular when moving from simple app to a non-trivial application? (Genuine question)

Inertia. The best practice for many reasons is to structure your app as lots of little components, and doing that in Angular is too hard, mostly because every component is at least three files, likely spread out in different places in the folder structure. The overhead of refactoring a component into smaller pieces is huge, since you have to independently split the template, the scope, and the dependencies.

All of the things that seem useful about Angular become a burden at scale. ng-repeat is my favorite example: try repeating more than a few hundred non-trivial components and you'll quickly choke the browser. You end up having to spend most of the application development time figuring out how to do things in a way that doesn't make Angular too upset.

Going from a simple single-controller to multiple controllers, services, and the rest was a hurdle I never managed to cross. I understand the principles involved, but actually going from "Here's my complete data model, handled by this controller" to "Here's an angular application with a number of controllers and services" isn't something I've managed to do. (Although this is also not something I've had to do professionally so it's not been a focus for me.)