I agree. XPath and freinds are too low level. Scrapy is nice though to control HTTP/assets download but in fact even this sucks. If you're serious with scraping you must go with a headless approach which also means Python-only doesn't work.