douwuli4512
douwuli4512
2016-05-26 19:57

通过Golang SDK将大文件发送到Amazon S3时如何使用较少的内存?

已采纳

I'm using AWS SDK on golang to send large backup files (2GB~10GB).

When the process starts there is a huge memory consumption. I know that this is because the code reads the file to a buffer, but I'm really new to Go, and I don't know how can I change this.

This is the code I'm using to read the file and send to AWS S3 Uploader:

file, err := os.Open(file_to_upload)

file_name := getFileName(file_to_upload)

if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
defer file.Close()

fileInfo, _ := file.Stat()
var size int64 = fileInfo.Size()

buffer := make([]byte, size)

file.Read(buffer)

fileBytes := bytes.NewReader(buffer)

fileType := http.DetectContentType(buffer)

upparams := &s3manager.UploadInput{
    Bucket: &bucket,
    Key:    &file_name,
    Body:   fileBytes,
    ACL:    aws.String("private"),
    ContentType:   aws.String(fileType),
    Metadata: map[string]*string{
        "Key": aws.String("MetadataValue"), //required
    },
}

result, err := uploader.Upload(upparams, func(u *s3manager.Uploader){
    if partsize != 0{
        u.PartSize = int64(partsize) * 1024 * 1024
    }
    u.LeavePartsOnError = false
    u.Concurrency = parallel
})

```

What I've tested so far.

Modification: Changed the u.Concurrency from 5 to 3: Outcome: CPU: Reduced the from 26% to 21% Memory: Same usage

Modification: Changed the u.Concurrency from 5 to 2: Outcome: CPU: Reduced the from 26% to 20% Memory: Same usage

Modification: Changed the u.Concurrency from 5 to 3 and u.Partsize to 100MB: Outcome: CPU: Reduced the from 26% to 16% Memory: Same usage

The time isn't the problem here but the memory consumption.

I want to tune this for the least resources possible. How can I approach to that?

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

1条回答

  • douguachi0056 douguachi0056 5年前

    There's no reason to read the entire file into memory. Just provide the Body field with the file itself

    upparams := &s3manager.UploadInput{
        Bucket: &bucket,
        Key:    &file_name,
    
        // *os.File is an io.Reader
        Body:   file,
    
        ACL:    aws.String("private"),
        ContentType:   aws.String(fileType),
        Metadata: map[string]*string{
            "Key": aws.String("MetadataValue"), //required
        },
    }
    
    点赞 评论 复制链接分享

为你推荐