在执行JavaScript后从dom获取html

Question

I may be saying this with incorrect terminology so correct me if I'm wrong please.

Here's what I want to do: I'm trying to scrape a website's comments section but the comments are loaded via an ajax call after the page has fully loaded. When I try to scrape the HTML from the site via:

res, err:= http.Get(url)
if err != nil {
    // handle error
}
defer res.Body.Close()

But it obviously gets the html before the ajax call. How do I go about getting the html after the ajax call?

This is completely off the top of my head, but would I need to basically create a js-renderer in code for this? My guess is that the JS needs to execute somehow. Any suggestions / libraries / examples on how to go about this? I'd prefer this to be in go but it could be realistically in any language.

Answer 1

If you own the site or can easily determine (or generate) the URI of the call that loads the comments, it's probably easier to make that same AJAX call yourself rather than bother with DOM parsing or arbitrary JS execution.

At that point Go would actually be a good language to use, since its JSON and XML standard libraries are excellent for unmarshalling that kind of data.