GoLang刮板机。 如何在网站上抓取动态生成的链接？

Question

I am trying to scrape product video links (which are generated dynamically by another web service. The location is under the product images on the left side). You can check following link, https://www.tokopedia.com/chocoapple/ready-stock-bnib-iphone-128gb-7-plus-jet-black-garansi-apple-1-tahun-10?src=topads The google chrome "inspect element" shows the div tag. But The same tag is not present in the page source. How to do it? I am looking into goQuery to implement the task but not sure will it work or not. I am not a web developer so please consider giving suggestions if my question description is not specific. Thank you.

Answer 1

If the tag is not in the source, then GoQuery will not work. GoQuery is for parsing HTML source using a jQuery-like API.

You need to first process the webpage with a headless WebKit like phantomjs, chromeless, or puppeteer. Each of these tools will allow you to process all the Javascript on the webpage before processing it. This way, the AJAX for rendering the video you are interest in will be processed and the source will be updated. You can then download the corresponding source which should have the div in it.