使用golang的http.ResponseWriter的AWS S3大文件反向代理

Question

I have a request handler named Download which I want to access a large file from Amazon S3 and push it to the user's browser. My goals are:

• To record some request information before granting the user access to the file
• To not buffer the file into memory too much. Files may become too large.

Here is what I've explored so far:

func Download(w http.ResponseWriter, r *http.Request) {

    sess := session.New(&aws.Config{
        Region:             aws.String("eu-west-1"),
        Endpoint:           aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:   aws.Bool(true),
        Credentials:        cred,
    })

    downloader := s3manager.NewDownloader(sess)
    // I can't write directly into the ResponseWriter. It doesn't implement WriteAt. 
    // Besides, it doesn't seem like the right thing to do.
    _, err := downloader.Download(w, &s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key: aws.String(filename),
    })
    if err != nil {
        log.Error(4, err.Error())
        return
    }

}

I'm wondering if there isn't a better approach (given the goals I'm trying to achieve).

Any suggestions are welcome. Thank you in advance :-)

Answer 1

If the file is potentially large, you don't want it to go trough your own server. The best approach (in my opinion) is to have the user download it directly from S3.

You can do this by generating a presigned url:

func Download(w http.ResponseWriter, r *http.Request) {

    ...

    sess := session.New(&aws.Config{
        Region:             aws.String("eu-west-1"),
        Endpoint:           aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:   aws.Bool(true),
        Credentials:        cred,
    })

    s3svc := s3.New(sess)
    req, _ := s3svc.GetObjectRequest(&s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key: aws.String(filename),
    })

    url, err := req.Presign(5 * time.Minute)
    if err != nil {
        //handle error
    }

    http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}

The presigned url is only valid for a limited time (5 minutes in this example, adjust to your needs) and takes the user directly to S3. No need to worry about downloads anymore!