从Google地图中抓取数据

Question

I am able to scrape the first 10 results of the maps directly through the default link, for example:

https://www.google.com/maps/search/toronto+dentist/

However I am unable to scrape the next 10 results because when I click on the right arrow, it uses Javascript to fetch the data, and there is no URL change for me to view the source of the next 10 results.

The classic Google maps version was scrapable, however the results differ from the current Google maps.

Answer 1

You want to use pjscrape. Instead of simply scraping the HTML, it has a full headless WebKit engine which can run the Javascript and render the page accordingly.

The API is bloody simple, just make sure that you rate-limit it because Google is quite fussy about that kind of thing.