从文件和存储文件创建sha256的最佳模式

Question

I am writing a webserver that receives a file as an upload as multipart/form-data. I am generating the file sha256 from the request but due to the nature of the Reader interface, I can't reuse the data to also upload the file to a filer. These files can be a few hundred MBs. What is the best way to store the content? I can duplicate the contents but I am worried that could be wasteful on memory resources.

EDIT

func uploadFile(w http.ResponseWriter, r *http.Request) {
    f, err := r.MultipartForm.File["capture"][0].Open()
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    defer f.Close()
    hash, err := createSha(f)
    if err != nil {
        fmt.Println(err.Error())
        return
    }
}

func createSha(image multipart.File) (hash.Hash, error) {
    sha := sha256.New()
    // This cause the contents of image to no longer be available to be read again to be stored on the filer
    if _, err := io.Copy(sha, image); err != nil {
        return nil, err
    }
    return sha, nil
}

Answer 1

You might use io.MultiWriter(...) to send the data to multiple output streams concurrently, such as a hash and some remote writer.

For example (roughly):

sha := sha256.New()
filer := filer.New(...) // Some Writer that stores the bytes for you?
err := io.Copy(io.MultiWriter(sha, filer), r)
// TODO: handle error
// Now sha.Sum(nil) has the file digest and "filer" got sent all the bytes.

Note that io.Multiwriter can take as many writers as you want, so you could compute additional hashes at the same time (e.g. md5, sha1, etc.) or even send the file to multiple locations, e.g.:

md5, sha1, sha256, sha512 := md5.New(), sha1.New(), sha256.New(), sha512.New()
s3Writer, gcsWriter := filer.NewS3Writer(), filer.NewGCSWriter()
mw := io.MultiWriter(awsWriter, gcsWriter, md5, sha1, sha256, sha512)
err := io.Copy(mw, r)
// TODO: handle error
// Now you've got all the hashes for the file and it's stored in the cloud.