How to do web scraping with Javascript

Hi I've encountered something i don't quite understand.

My objective was to scrape some data from every product in a game's marketplace (thetan arena). The thing is, when i try to get something to lead me to the product's url, i always get to the image's url or every other useless information but never something to get me to the actual page of the product.

For instance... when i right click on the product, there's no option from the browser to copy link or open in a new tab.

So i think i can't do it using DOM methods, right?

Any advice? Kinda super stuck here =(

Thanks for your attention, merry xmas :)

TuqueroIvantckJernalyn

4 years ago

Level 1

Ive done a few commercial web scrapers using javascript, and as far as scraping technology goes - its among my top picks. However, a lot of the baggage associated with JS rears its ugly head for this use case. Web scrapers are almost always meant to be procedural and error proof. JS is bad at both. For some projects, such as dynamically rendered sites - you need JS or https://chatspin.download you need a reverse proxy. Setting up a proxy for a https://echat.date quick job is overkill so thats when I would lean on JS. NightmareJS was my library of choice but these days Id use Pupeteer.

Please or to participate in this conversation.