I think you want to have a system that can use XPath or CSS queries to select the elements you want.
This way writing a scraper for a given page is almost as easy as right clicking on an element in dev tools and selecting "Copy as XPath" for what you want.
You definitely need some validation that your scraper is still returning accurate results, so that you can get notified when things go wrong. Things like following links from an item to the item's product page and comparing scraped prices, names & images should get you a lot of the way.
At some point this will definitely get unwieldy, and you can try to build a more general solution that can understand grids or layout, but despite my preference for this as both a shopper of long tail sites and a developer, this is probably not where you want to start unless the long tail is your actual niche.
You recommend staying specific to few categories instead of crawling over everything available on internet? We are starting only with women's clothing.
I was more referencing the typical approach that people took of supporting the top N most popular sites and increasing N as they got bigger.
It's a solid approach for hitting the majority of the market, and works fine for alerting, but this leaves a pretty big gap in the market for people who are interested in comparison shopping for more boutique items, e.g. designer male fashion get sold by piles of different boutiques, each with their own sales, etc, but the items are exactly the same, and I would really like to know when something I am interested in goes on sale at one of the 50 different stores that have this item, and I would like to know this only when they go on sale in my size, and whether it's actually cheap after currency conversion and shipping. A person can dream, right?
Shit, I would love it if there was a platform that could guess my size across various items in different brands.
I've thought of this space a bit since I buy a decent amount of clothes, but I've never gone ahead and tried to execute.
This way writing a scraper for a given page is almost as easy as right clicking on an element in dev tools and selecting "Copy as XPath" for what you want.
You definitely need some validation that your scraper is still returning accurate results, so that you can get notified when things go wrong. Things like following links from an item to the item's product page and comparing scraped prices, names & images should get you a lot of the way.
At some point this will definitely get unwieldy, and you can try to build a more general solution that can understand grids or layout, but despite my preference for this as both a shopper of long tail sites and a developer, this is probably not where you want to start unless the long tail is your actual niche.