dry18813
2018-03-14 17:36
浏览 72
已采纳

Snowball Edge-Golang中的aws-sdk-go软件包-无法连接到S3

I'm using the aws-sdk-go package in Golang to connect to Amazon S3 to provide a cloud-based storage pool. I have this working well. I would like to be able to support bulk high-speed transfers using Snowball, so I got a Snowball Edge to test this in my lab. I have not figured out how to get this working, and the documentation for Snowball Edge doesn't seem complete. This configuration may be impacted by having ordered a Snowball Edge and not just a Snowball.

The reason that I'm finding the Edge more problematic, is that a normal Snowball requires an application called snowballAdapter to be running, which looks like it handles some port mapping issues. But, this application seems to be incompatible with the Edge device, as it reports that it doesn't work with a "Snowball Edge Manifest file".

I looked at the ports that are available on the real AWS S3 and nmap reports:

nmap -v -sT -Pn s3.us-east-1.amazonaws.com
...
Scanning s3.us-east-1.amazonaws.com (52.216.161.53) [1000 ports]
Discovered open port 443/tcp on 52.216.161.53
Discovered open port 80/tcp on 52.216.161.53

Whereas on Snowball Edge, the ports are:

nmap -v -sT -Pn 192.168.1.4
....
Scanning 192.168.1.4 [1000 ports]
Discovered open port 8080/tcp on 192.168.1.4
Discovered open port 22/tcp on 192.168.1.4
Discovered open port 9091/tcp on 192.168.1.4
Discovered open port 8443/tcp on 192.168.1.4
....
PORT     STATE SERVICE
22/tcp   open  ssh
8080/tcp open  http-proxy
8443/tcp open  https-alt
9091/tcp open  xmltec-xmlmail

So, it seems to me that the issue may be that I have to make the aws package use port 8443 for Snowball Edge instead of 443 for the real S3. The code for connecting to S3 is pretty straight forward:

creds := credentials.NewStaticCredentials(s3Config.S3AccessKey, s3Config.S3SecretAccessKey, s3Config.S3Token)
_, err := creds.Get()

if err != nil {
    return nil, nil, err
}

if len(baseFolder) > 0 {
    baseFolder = baseFolder + "/"
}

cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds)
svc := s3.New(session.New(), cfg)

params := &s3.ListObjectsInput{
    Bucket:    aws.String(s3Config.S3BucketName),
    Prefix:    aws.String(baseFolder),
    Delimiter: aws.String("/"),
}

resp, err := svc.ListObjects(params)

So, the question is, how do I change the code to point to the Snowball Edge? I've tried mapping to the Snowball Edge in /etc/hosts from the Amazon S3 endpoint. I understand why this didn't work after discovering that the ports are different. I've played around with adding different forms of "WithEndpoint("...host...") with no success. Or, am I on the completely wrong track, and should be able to get the snowballAdapter to work with a Snowball Edge?

By the way, all the snowballEdge commands work as expected, so the device seems to be working fine, e.g.:

./snowballEdge list-access-keys

    {
      "AccessKeyIds" : [ "..." ]
    }

./snowballEdge get-secret-access-key --access-key-id ....

    [snowballEdge]
    aws_access_key_id = ...
    aws_secret_access_key = ...

And, I've used the correct keys associated with the device, and it does have the S3 Service configured in it:

./snowballEdge list-services
{
  "ServiceIds" : [ "s3" ]
}
  • 写回答
  • 好问题 提建议
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • duankezong4064 2018-05-08 18:59
    已采纳

    Snowball Edge is a very different beast than is AWS S3. In addition to the access key and secret access key, it requires that an endpoint and additional credentials in the form of a certificate. The real AWS S3 has a valid certificate, supported by the certificate authorities, but Snowball Edge has a self-signed certificate.

    The configuration can be created with this:

    cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds).WithEndpoint(s3Config.S3Endpoint).WithHTTPClient(httpClient)
    svc = s3.New(session.New(), cfg)
    

    The endpoint looks like this (you see Snowball Edge responding on port 8443 in the question text above):

    https://192.168.1.4:8443
    

    The certificate is required to be formatted like you'd have in a file (including all the new line characters. It looks like this (again, the string you pass in has to include the new line characters after every line):

    -----BEGIN CERTIFICATE-----
    MIIC7zCCAdegAwIBAgIJBJZB/gkBP0B5MA0GCSqGSIb3DQEBCwUAMBYxFDASBgNV
    BAMMCzE3Mi4yMC4xLjE3NB4XDTE3MDQyODIwMTMwOFoXDTE5MDQxODIwMTMwOFow
    FjEUMBIGA1UEAwwLMTcyLjIwLjEuMTcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
    ggEKAoIBAQCrLWlTfrSj9R1of5Z98EHYIEEPBgnWxnlrvA+ryAzPmiXbYomI4Tpl
    PsuIA+7hGXG10H0zwlz0n22EUv4pE79toYcd3czOJUAHEuSelhtP7u91vM4GguFx
    A00gosu04RFUD+BYNeaLTQfd7vdmQB3bY3KEbn7Dfrs/1MYFhKb8J77mgCuUbAPu
    PNvwLoV+hBL+ndgs+bIu4MtXjUJDiigRZkacpMQaduDMqEq6seoc+JwrKNBjRBRu
    3l/fcQoWf+g902oZJaXXnVGqqb7o2YAQFehUAbmCJfuKFSl5tu0B+3KvQQni7lK+
    SV8WItdrPumS98BBlt6NpzgC5fTwCmapAgMBAAGjQDA+MAwGA1UdEwQFMAMBAf8w
    HQYDVR0OBBYEFMAvKzKgKI+izqPX6DJjJz/0fELtMA8GA1UdEQQIMAaHBKwUAREw
    DQYJKoZIhvcNAQELBQADggEBAGwyzmI+9psQu9/N/oClN7Lej7e4E8cC8vymVfPz
    fdW45IMNVEYHxHbu9+JzLOtLxfuDmD6B6fEYVoPubb6tsnacuwOSMZhTvhhy9nv2
    f+2Pslgj/kYTeMePbHOPTyQ4sd1BE7ALdNiL/hd08ZNhqObagixNYw9eYeHEStBy
    tOADKcY9gOxek1k9t+96nATgSy0WIytwra0uEgyipKQ2gXKpgg15SI4nDxQLLEgG
    lb3FtRk+PfJxQ4zbHZe/cRNflcGwVCefycLQOA2Sdr8pgHW7gvETu9i9ywF0UV6f
    b9wsPcDmg3EaxBa+wrLlYSzaPhI+rZYh6bpnTn311QIFZ+s=
    -----END CERTIFICATE-----
    

    Found the Snowball Edge to be pretty fast storage using S3. We were able to achieve about 1,100 MBytes/second of copy rate.

    已采纳该答案
    评论
    解决 无用
    打赏 举报

相关推荐 更多相似问题