无法使用goquery收集Google搜索结果的所有节点：缺少某些节点

Question

I am trying to collect results of a google search page in GoLang using the goquery library. In order to achieve this, I am collecting all nodes of a goquery selection with goquery. The problem is that the selection returned by Find("*") does not seem to contain all the nodes of the HTML document. Question: does the method collect ALL nodes with the whole tree structure or not ? If not, is there a method to collect them all ?

I tried using the goquery Find("*") method applied to the whole document selection. So nodes with certain attributes are not returned, although they are in the HTML document. For instance, nodes with are not recognized

alltags := doc.Find("*") //doc is the HTML doc with the Google search

The selection does not contain the div tags with class="srg". The same applies to other class values such as "bkWMgd", "rc" for example.

Answer 1

This has happened to me before. I was trying to web scrape with python beautiful soup package and the same thing was happening.

Later it turned out that the html markup returned when trying to fetch it was actually the markup the server returned after finding a bot. I solved this by setting the User-Agent to Mozilla/5.0.

Hope this helps in your quest to solve this. You can start by updating the code for the fetch request you have performed.