如何在Golang中有效存储对文件的HTML响应

Question

I'm trying to build a crawler in Golang. I'm using net/http library to download the html file from url. I'm trying to save http.resp and http.Header into file.

How to convert these two file from their respective format into string so that, it could be written to a text file.

I also see a question asked earlier on parsing a stored html response file. Parse HTTP requests and responses from text file in Go . Is there any way to save the url response in this format.

Answer 1

Edit: Thanks to @JimB for pointing to the http.Response.Write method which makes this a lot easier than I proposed in the beginning:

resp, err := http.Get("http://google.com/")

if err != nil{
    log.Panic(err)
}

f, err := os.Create("output.txt")
defer f.Close()

resp.Write(f)

This was my first Answer

You could do something like this:

resp, err := http.Get("http://google.com/")

body, err := ioutil.ReadAll(resp.Body)

// write whole the body
err = ioutil.WriteFile("body.txt", body, 0644)
if err != nil {
    panic(err)
}

This was the edit to my first answer:

Thanks to @Hector Correa who added the header part. Here is a more comprehensive snippet, targeting your whole question. This writes header followed by the body of the request to output.txt

//get the response
resp, err := http.Get("http://google.com/")

//body
body, err := ioutil.ReadAll(resp.Body)

//header
var header string
for h, v := range resp.Header {
    for _, v := range v {
        header += fmt.Sprintf("%s %s 
", h, v)
    }
}

//append all to one slice
var write []byte
write = append(write, []byte(header)...)
write = append(write, body...)

//write it to a file
err = ioutil.WriteFile("output.txt", write, 0644)
if err != nil {
    panic(err)
}