douren7921 2017-04-24 19:06
浏览 155

如何在S3中保存数据流? aws-sdk-go示例不起作用?

I am trying to persist a given stream of data to an S3 compatible storage. The size is not known before the stream ends and can vary from 5MB to ~500GB.

I tried different possibilities but did not find a better solution than to implement sharding myself. My best guess is to make a buffer of a fixed size fill it with my stream and write it to the S3. Is there a better solution? Maybe a way where this is transparent to me, without writing the whole stream to memory?

The aws-sdk-go readme has an example programm that takes data from stdin and writes it to S3:

When I try to pipe data in with a pipe | I get the following error: failed to upload object, SerializationError: failed to compute request body size caused by: seek /dev/stdin: illegal seek Am I doing something wrong or is the example not working as I expect it to?

I although tried minio-go, with PutObject() or client.PutObjectStreaming(). This is functional but consumes as much memory as the data to store.

  1. Is there a better solution?
  2. Is there a small example program that can pipe arbitrary data into S3?
  • 写回答

1条回答 默认 最新


      相关推荐 更多相似问题


      • ¥15 结构体数组文件读取问题
      • ¥15 写c++代码,第7题说一下使用的计算方法
      • ¥15 postman接口自动化测试报告实践总结
      • ¥15 有关c++的问题,利用相关知识
      • ¥15 求香农编码和解码的matlab代码
      • ¥20 ROS中的TEB局部规划问题
      • ¥20 关于#matlab#的问题:要求测出上面图片中所有同心圆的半径
      • ¥20 epanet软件运行问题
      • ¥15 Python 文件读取
      • ¥60 dpabi进行Alff计算时脑池有干扰信号