从网站抓取Ajax内容

Question

There is a site researcherid. On author page, for example, http://www.researcherid.com/rid/A-7364-2016 i need to retrieve information from Citation metrics window. That`s the window generated by ajax. How can i do it with Python ? What tools i have for it? I tried:

payload = {'key': '721889',
       'listid': 'LIST1',
       'displayName': 'My Publications', 'publicProfile': 'true', '_': ''
       }

r = requests.post('http://www.researcherid.com/rid/A-7364-2016',
              data=payload, headers={
                  'X-Requested-With': 'XMLHttpRequest'
              })
print(r.text)

But it does`t work.

Answer 1

You can use Selenium's webdriver tools in order to get dynamically loaded content. Simply create a webdriver instance, go to Citation metrics page and load source into python. After that, you can scrape content with libraries such as LXML or BeautifulSoup.