Level 18
Goutte maybe https://github.com/FriendsOfPHP/Goutte
Nov 16, 2015
7
Level 23
Scrape Ajax Loaded Content of external site
Hi,
I need to get some data from an external site. That site loads the content with js, so CURL won't work.
I've manage to do it using selenium on my local computer. But on my server I can't use java, hence I can't use selenium.
Is there any other way to fetch the html from a site that loads its content with js?
Thanks, Diogo
Level 28
Use the Network tab in Chrome's dev tools to find the endpoint the site is loading their data via JS, and then try using CURL to reach that endpoint.
Level 47
What are you doing with the data? Making a call to save it in your database or just displaying the data?
Level 23
@jekinney I'm parsing the html, to fetch some data, and then save it to the database.
Level 28
This is written for Python, but the part relating to finding the AJAX request using Chrome's dev tools should help.
Level 47
@diugo your getting the full html page? Not a API with Json or XML?
Reason I am asking there are a few different ways to do this. It can be done with php (file_get_contents) and a few other ways. Also Javascript can scrape sites too..
Level 23
@jekinney I just want the html (I can parse the data after that).
On this page the html is loaded with javascript. So file_get_contents, cURL,... will get empty html (just the base structure) not that data.
I've managed to do it with casperjs and slimerjs on my local computer. The problem now is that I can't make them work on a shared hosting service like Hostgator.
Cheers
Please or to participate in this conversation.