douling8087 2019-07-14 08:53
浏览 91
已采纳

Ajax加载网站内容后进行Web爬取

I'm trying to get colly to scrape the following page: https://www56.muenchen.de/termin/index.php?loc=BB.

Here is my code:

package main

import (
    "fmt"
    "log"

    "github.com/gocolly/colly"
)

func main() {
    c := colly.NewCollector(
        colly.IgnoreRobotsTxt(),
        colly.Async(false),
    )

    c.OnHTML("html", func(e *colly.HTMLElement) {
        fmt.Println(e.Text)
    })

    c.OnError(func(_ *colly.Response, err error) {
        log.Println("Something went wrong:", err)
    })

    c.Visit("https://www56.muenchen.de/termin/index.php?loc=BB")

    c.OnScraped(func(r *colly.Response) {
        fmt.Println("Finished")
    })
}

The problem is that after the website is visited it loads some content. I'm unsure how to tell colly to "wait" until that has happened and then look at the result.

Looking forward to some ideas.

展开全部

  • 写回答

1条回答 默认 最新

  • douzhouhan4618 2019-07-14 10:11
    关注

    It can't since colly would have to do that client-side, but colly does not execute JavaScript - so no Ajax with it.

    To simulate a browser you can use selenium or phantomjs as the link above suggests.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
编辑
预览

报告相同问题?

悬赏问题

  • ¥15 PADS Logic 原理图
  • ¥15 PADS Logic 图标
  • ¥15 电脑和power bi环境都是英文如何将日期层次结构转换成英文
  • ¥20 气象站点数据求取中~
  • ¥15 如何获取APP内弹出的网址链接
  • ¥15 wifi 图标不见了 不知道怎么办 上不了网 变成小地球了
手机看
程序员都在用的中文IT技术交流社区

程序员都在用的中文IT技术交流社区

专业的中文 IT 技术社区,与千万技术人共成长

专业的中文 IT 技术社区,与千万技术人共成长

关注【CSDN】视频号,行业资讯、技术分享精彩不断,直播好礼送不停!

关注【CSDN】视频号,行业资讯、技术分享精彩不断,直播好礼送不停!

客服 返回
顶部