2018-03-14 17:36
浏览 72

Snowball Edge-Golang中的aws-sdk-go软件包-无法连接到S3

I'm using the aws-sdk-go package in Golang to connect to Amazon S3 to provide a cloud-based storage pool. I have this working well. I would like to be able to support bulk high-speed transfers using Snowball, so I got a Snowball Edge to test this in my lab. I have not figured out how to get this working, and the documentation for Snowball Edge doesn't seem complete. This configuration may be impacted by having ordered a Snowball Edge and not just a Snowball.

The reason that I'm finding the Edge more problematic, is that a normal Snowball requires an application called snowballAdapter to be running, which looks like it handles some port mapping issues. But, this application seems to be incompatible with the Edge device, as it reports that it doesn't work with a "Snowball Edge Manifest file".

I looked at the ports that are available on the real AWS S3 and nmap reports:

nmap -v -sT -Pn s3.us-east-1.amazonaws.com
Scanning s3.us-east-1.amazonaws.com ( [1000 ports]
Discovered open port 443/tcp on
Discovered open port 80/tcp on

Whereas on Snowball Edge, the ports are:

nmap -v -sT -Pn
Scanning [1000 ports]
Discovered open port 8080/tcp on
Discovered open port 22/tcp on
Discovered open port 9091/tcp on
Discovered open port 8443/tcp on
22/tcp   open  ssh
8080/tcp open  http-proxy
8443/tcp open  https-alt
9091/tcp open  xmltec-xmlmail

So, it seems to me that the issue may be that I have to make the aws package use port 8443 for Snowball Edge instead of 443 for the real S3. The code for connecting to S3 is pretty straight forward:

creds := credentials.NewStaticCredentials(s3Config.S3AccessKey, s3Config.S3SecretAccessKey, s3Config.S3Token)
_, err := creds.Get()

if err != nil {
    return nil, nil, err

if len(baseFolder) > 0 {
    baseFolder = baseFolder + "/"

cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds)
svc := s3.New(session.New(), cfg)

params := &s3.ListObjectsInput{
    Bucket:    aws.String(s3Config.S3BucketName),
    Prefix:    aws.String(baseFolder),
    Delimiter: aws.String("/"),

resp, err := svc.ListObjects(params)

So, the question is, how do I change the code to point to the Snowball Edge? I've tried mapping to the Snowball Edge in /etc/hosts from the Amazon S3 endpoint. I understand why this didn't work after discovering that the ports are different. I've played around with adding different forms of "WithEndpoint("...host...") with no success. Or, am I on the completely wrong track, and should be able to get the snowballAdapter to work with a Snowball Edge?

By the way, all the snowballEdge commands work as expected, so the device seems to be working fine, e.g.:

./snowballEdge list-access-keys

      "AccessKeyIds" : [ "..." ]

./snowballEdge get-secret-access-key --access-key-id ....

    aws_access_key_id = ...
    aws_secret_access_key = ...

And, I've used the correct keys associated with the device, and it does have the S3 Service configured in it:

./snowballEdge list-services
  "ServiceIds" : [ "s3" ]
  • 写回答
  • 好问题 提建议
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • duankezong4064 2018-05-08 18:59

    Snowball Edge is a very different beast than is AWS S3. In addition to the access key and secret access key, it requires that an endpoint and additional credentials in the form of a certificate. The real AWS S3 has a valid certificate, supported by the certificate authorities, but Snowball Edge has a self-signed certificate.

    The configuration can be created with this:

    cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds).WithEndpoint(s3Config.S3Endpoint).WithHTTPClient(httpClient)
    svc = s3.New(session.New(), cfg)

    The endpoint looks like this (you see Snowball Edge responding on port 8443 in the question text above):

    The certificate is required to be formatted like you'd have in a file (including all the new line characters. It looks like this (again, the string you pass in has to include the new line characters after every line):

    -----END CERTIFICATE-----

    Found the Snowball Edge to be pretty fast storage using S3. We were able to achieve about 1,100 MBytes/second of copy rate.

    解决 无用
    打赏 举报

相关推荐 更多相似问题