在cURL下载页面之前等待Javascript？

Question

How does one wait until all of the Javascript is loaded before curling a website? I am trying to download the HTML from one of my pages, but it fetches information asynchronously, so cURL fetches a half loaded page. Is there a way to get cURL to fetch a fully loaded page?

Answer 1

You need to use a headless browser engine to do this. cURL and wget are HTTP libraries; they speak HTTP and download documents as text. They don't have a concept of a DOM or a JavaScript engine that would help them understand that a page is doing AJAX. So to download the HTML, you need something that acts more like a browser, by parsing a DOM and executing JS. I recommend Crowbar, which uses a Mozilla engine.