为什么我的代码花这么长时间才能返回结果?

When running this code I have to wait 10 seconds for s.Locations to print and 60+ seconds for n.Titles to print. What is causing this?

Tips on how to troubleshoot this would be helpful i.e. seeing how long it takes for certain lines of code to complete. New to Go so not sure how to exactly do this.

I've made sure I close my connections. Since everything else on my computer loads blazing fast I don't think to access the internet via http.Get should be slow.

package main

import (
    "encoding/xml"
    "fmt"
    "io/ioutil"
    "net/http"
    "strings"
)

// SitemapIndex is the root xml
type SitemapIndex struct {
    Locations []string `xml:"sitemap>loc"`
}

// News is the individual categories
type News struct {
    Titles    []string `xml:"url>news>title"`
    Keywords  []string `xml:"url>news>keywords"`
    Locations []string `xml:"url>loc"`
}

// NewsMap is the
type NewsMap struct {
    Keywords string
    Location string
}

func main() {
    var s SitemapIndex
    var n News
    // np := make(map[string]NewsMap)
    resp, _ := http.Get("https://www.washingtonpost.com/news-sitemaps/index.xml")
    bytes, _ := ioutil.ReadAll(resp.Body)
    xml.Unmarshal(bytes, &s)
    resp.Body.Close()

    for i := range s.Locations {
        s.Locations[i] = strings.TrimSpace(s.Locations[i])
    }

    fmt.Println(s.Locations) // slice of data

    for _, Location := range s.Locations {
        resp, _ := http.Get(Location)
        bytes, _ := ioutil.ReadAll(resp.Body)
        xml.Unmarshal(bytes, &n)
        resp.Body.Close()
    }

    fmt.Println(n.Titles)
}

I get the output but I have to wait 10 seconds for s.Locations and 60+ seconds for n.Titles

duanbiaojin8860
duanbiaojin8860 与使用浏览器获取页面相比,此程序和curl的速度较慢。服务器可能会限制对站点的自动访问。
大约一年之前 回复
dragonlew9876
dragonlew9876 正在编辑我的代码。从@CeriseLimón找到了一篇有关如何计时执行时间的帖子。我喜欢深入了解如何做。谢谢你让我思考。
大约一年之前 回复
dongmei425373
dongmei425373 “也许吧”---测量它。
大约一年之前 回复
douhuang2673
douhuang2673 也许我没有,但这就是我想发生的事情。请纠正我。第一个http.Get加载index.xml页,并以字节为单位读取所有数据。xml.Unmarshal将这些字节转换为字符串,然后将与xml:sitemap>loc匹配的Location保存到数组的切片中。因此,也许将所有xml转换为字节,然后将它们打印到屏幕上才是瓶颈,而不是12个http.Get调用?
大约一年之前 回复
dongwu8653
dongwu8653 啊,你说得对。我忘记了它解析的其他子类别。
大约一年之前 回复
dongzhao2725
dongzhao2725 “我了解它的客户端速度很慢”---您为什么这么认为?
大约一年之前 回复
dqdes60666
dqdes60666 该应用程序总共发出12个请求。服务器响应缓慢。
大约一年之前 回复
dongliu4320
dongliu4320 有什么办法可以优化吗?我只发出2个get请求,所以我担心在添加代码,更多http请求和扩展时会发生什么。我了解客户端很慢。
大约一年之前 回复
donglu8334
donglu8334 令您感到惊讶的并不明显。这需要一些时间,因为您需要发送http请求并获得响应。
大约一年之前 回复
dqnk57224
dqnk57224 你能解释一下吗?我正在跟踪教程,因此该代码不是我的原始代码。另外,我没有任何错误。
大约一年之前 回复

1个回答

Tips on how to troubleshoot this would be helpful.


Start with the simple things, measuring one thing at a time, a scientific experiment.


Use curl to measure basic response time.

$ curl https://www.google.com/robots.txt -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7246    0  7246    0     0  94103      0 --:--:-- --:--:-- --:--:-- 94103
$ curl https://www.nytimes.com/robots.txt -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   934  100   934    0     0     61      0  0:00:15  0:00:15 --:--:--   230
$ curl https://www.washingtonpost.com/robots.txt -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3360  100  3360    0     0    133      0  0:00:25  0:00:25 --:--:--   869
$

Google has no delay. The New York Times has a 15 second delay. The Washington Post has a 25 second delay.

In Go, confirm that The Washington Post has a 25 second delay.

$ go run wapo.go
25.174366651s
$ cat wapo.go
package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "time"
)

func main() {
    start := time.Now()
    resp, err := http.Get("https://www.washingtonpost.com/news-sitemaps/index.xml")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    _, err = ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    resp.Body.Close()
    fmt.Fprintln(os.Stderr, time.Since(start))
}
$

Next, try a different ISP from a different computer.

$ curl https://www.google.com/robots.txt -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7246    0  7246    0     0  27343      0 --:--:-- --:--:-- --:--:-- 27343
$ curl https://www.nytimes.com/robots.txt -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   934  100   934    0     0   2017      0 --:--:-- --:--:-- --:--:--  2017
$ curl https://www.washingtonpost.com/robots.txt -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3360  100  3360    0     0    356      0  0:00:09  0:00:09 --:--:--   840
$ curl https://www.washingtonpost.com/news-sitemaps/index.xml -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1101  100  1101    0     0    104      0  0:00:10  0:00:10 --:--:--   266
$ 

$ go run wapo.go
8.378458882s
$ 

Google has no delay. The New York Times has a small delay. The Washington Post has a 9 second delay.


The Go code and compiler are the same:

$ go version
go version devel +a25c2878c7 Sat Jul 27 23:29:18 2019 +0000 linux/amd64
$ cat wapo.go
package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

func main() {
    start := time.Now()
    resp, err := http.Get("https://www.washingtonpost.com/news-sitemaps/index.xml")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    resp.Body.Close()
    fmt.Fprintln(os.Stderr, time.Since(start))
}
$

Therefore, focus on network and site factors.


duandi8613
duandi8613 对其他人的站点/服务进行负载测试的形式不正确,有充分的理由会使您的IP被阻止。 不要这样
大约一年之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问
相关内容推荐