如何在AWS开发工具包中实现AWS CLI Sync Command的性能

The aws s3 sync command in the CLI can download a large collection of files very quickly, and I can not achieve the same performance with the AWS Go SDK. I have millions of files in the bucket so this is critical to me. I need to use the list pages command as well so that I can add a prefix which is not supported well by the sync CLI command.

I have tried using multiple goroutines (10 up to 1000) to make requests to the server, but the time is just so much slower compared to the CLI. It takes about 100 ms per file to run the Go GetObject function which is unacceptable for the number of files that I have. I know that the AWS CLI also uses the Python SDK in the backend, so how does it have so much better performance (I tried my script in boto as well as Go).

I am using ListObjectsV2Pages and GetObject. My region is the same as the S3 server's.

    logMtx := &sync.Mutex{}
    logBuf := bytes.NewBuffer(make([]byte, 0, 100000000))

    err = s3c.ListObjectsV2Pages(
        &s3.ListObjectsV2Input{
            Bucket:  bucket,
            Prefix:  aws.String("2019-07-21-01"),
            MaxKeys: aws.Int64(1000),
        },
        func(page *s3.ListObjectsV2Output, lastPage bool) bool {
            fmt.Println("Received", len(page.Contents), "objects in page")
            worker := make(chan bool, 10)
            for i := 0; i < cap(worker); i++ {
                worker <- true
            }
            wg := &sync.WaitGroup{}
            wg.Add(len(page.Contents))
            objIdx := 0
            objIdxMtx := sync.Mutex{}
            for {
                <-worker
                objIdxMtx.Lock()
                if objIdx == len(page.Contents) {
                    break
                }
                go func(idx int, obj *s3.Object) {
                    gs := time.Now()
                    resp, err := s3c.GetObject(&s3.GetObjectInput{
                        Bucket: bucket,
                        Key:    obj.Key,
                    })
                    check(err)
                    fmt.Println("Get: ", time.Since(gs))

                    rs := time.Now()
                    logMtx.Lock()
                    _, err = logBuf.ReadFrom(resp.Body)
                    check(err)
                    logMtx.Unlock()
                    fmt.Println("Read: ", time.Since(rs))

                    err = resp.Body.Close()
                    check(err)
                    worker <- true
                    wg.Done()
                }(objIdx, page.Contents[objIdx])
                objIdx += 1
                objIdxMtx.Unlock()
            }
            fmt.Println("ok")
            wg.Wait()
            return true
        },
    )
    check(err)

Many results look like:

Get:  153.380727ms
Read:  51.562µs
查看全部
dpd7195
dpd7195
2019/07/22 21:55
  • it技术
  • 互联网问答
  • IT行业问题
  • 编程语言问答
  • 计算机技术
  • 点赞
  • 收藏
  • 回答
    私信
满意答案
查看全部

2个回复