如何在Golang中不使用ioutil.ReadAll检索网站源代码

Question

My code:

func getSourceUrl(url string) (string, error) {
   resp, err := http.Get(url)
   if err != nil {
       fmt.Println("Error getSourceUrl: ")
       return "", err
   }
   defer resp.Body.Close()
   body := resp.Body
   // time = 0
   sourcePage, err := ioutil.ReadAll(body)
   // time > 5 minutes
   return string(sourcePage), err
}

I have a website link with a source of around> 100000 lines. Using ioutil.ReadAll made me get very long (about> 5 minutes for 1 link). Is there a way to get Source website faster? Thank you!

Answer 1

You could iterate sections of the response at a time, something like;

responseSection := make([]byte, 128)
body.Read(responseSection)
return string(responseSection), err

Which would read 128 bytes at a time. However would suggest confirming the download speed is not causing the slow load.