douwanc63652
2015-12-09 11:01
浏览 840
已采纳

使用go将文件流传输到AWS S3

I want to stream a multipart/form-data (large) file upload directly to AWS S3 with as little memory and file disk footprint as possible. How can I achieve this? Resources online only explain how to upload a file and store it locally on the server.

图片转代码服务由CSDN问答提供 功能建议

我想用较少的内存和数据直接将多部分/表单数据(大)文件流传输到AWS S3 文件磁盘占用的空间。 我该如何实现? 在线资源仅说明如何上传文件并将其本地存储在服务器上。

  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

5条回答 默认 最新

  • duangongqiong6958 2015-12-21 14:31
    已采纳

    you can do this using minio-go :

    n, err := s3Client.PutObject("bucket-name", "objectName", object, size, "application/octet-stream")
    

    PutObject() automatically does multipart upload internally. Example

    打赏 评论
  • dongsha9208 2015-12-09 11:48

    I didn't try it but if i were you id try the multi part upload option .

    you can read the doc multipartupload .

    here is go example for multipart upload and multipart upload abort.

    打赏 评论
  • dongmu6578 2015-12-12 08:28

    Amazon has an official Go package for uploading files to S3.

    http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/Uploader.html

    They also have an example of uploading a file while zipping it on the fly.

    https://github.com/aws/aws-sdk-go/wiki/common-examples#upload-an-arbitrarily-sized-stream-with-amazon-s3-upload-manager

    Not sure if that'll help. Your question was kinda vague.

    打赏 评论
  • doujiao1981 2015-12-15 23:01

    Another option is to mount the S3 bucket with goofys and then stream your writes to the mountpoint. goofys does not buffer the content locally so it will work fine with large files.

    打赏 评论
  • duanping1632 2019-04-02 23:03

    You can use upload manager to stream the file and upload it, you can read comments in source code you can also configure params to set the part size, concurrency & max upload parts, below is a sample code for reference.

    package main
    
    import (
        "fmt"
        "os"
    
        "github.com/aws/aws-sdk-go/aws/credentials"
    
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/s3/s3manager"
    )
    
    var filename = "file_name.zip"
    var myBucket = "myBucket"
    var myKey = "file_name.zip"
    var accessKey = ""
    var accessSecret = ""
    
    func main() {
        var awsConfig *aws.Config
        if accessKey == "" || accessSecret == "" {
            //load default credentials
            awsConfig = &aws.Config{
                Region: aws.String("us-west-2"),
            }
        } else {
            awsConfig = &aws.Config{
                Region:      aws.String("us-west-2"),
                Credentials: credentials.NewStaticCredentials(accessKey, accessSecret, ""),
            }
        }
    
        // The session the S3 Uploader will use
        sess := session.Must(session.NewSession(awsConfig))
    
        // Create an uploader with the session and default options
        //uploader := s3manager.NewUploader(sess)
    
        // Create an uploader with the session and custom options
        uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
            u.PartSize = 5 * 1024 * 1024 // The minimum/default allowed part size is 5MB
            u.Concurrency = 2            // default is 5
        })
    
        //open the file
        f, err := os.Open(filename)
        if err != nil {
            fmt.Printf("failed to open file %q, %v", filename, err)
            return
        }
        //defer f.Close()
    
        // Upload the file to S3.
        result, err := uploader.Upload(&s3manager.UploadInput{
            Bucket: aws.String(myBucket),
            Key:    aws.String(myKey),
            Body:   f,
        })
    
        //in case it fails to upload
        if err != nil {
            fmt.Printf("failed to upload file, %v", err)
            return
        }
        fmt.Printf("file uploaded to, %s
    ", result.Location)
    }
    
    打赏 评论

相关推荐 更多相似问题