dry18813 2018-03-14 17:36
浏览 73
已采纳

Snowball Edge-Golang中的aws-sdk-go软件包-无法连接到S3

I'm using the aws-sdk-go package in Golang to connect to Amazon S3 to provide a cloud-based storage pool. I have this working well. I would like to be able to support bulk high-speed transfers using Snowball, so I got a Snowball Edge to test this in my lab. I have not figured out how to get this working, and the documentation for Snowball Edge doesn't seem complete. This configuration may be impacted by having ordered a Snowball Edge and not just a Snowball.

The reason that I'm finding the Edge more problematic, is that a normal Snowball requires an application called snowballAdapter to be running, which looks like it handles some port mapping issues. But, this application seems to be incompatible with the Edge device, as it reports that it doesn't work with a "Snowball Edge Manifest file".

I looked at the ports that are available on the real AWS S3 and nmap reports:

nmap -v -sT -Pn s3.us-east-1.amazonaws.com
...
Scanning s3.us-east-1.amazonaws.com (52.216.161.53) [1000 ports]
Discovered open port 443/tcp on 52.216.161.53
Discovered open port 80/tcp on 52.216.161.53

Whereas on Snowball Edge, the ports are:

nmap -v -sT -Pn 192.168.1.4
....
Scanning 192.168.1.4 [1000 ports]
Discovered open port 8080/tcp on 192.168.1.4
Discovered open port 22/tcp on 192.168.1.4
Discovered open port 9091/tcp on 192.168.1.4
Discovered open port 8443/tcp on 192.168.1.4
....
PORT     STATE SERVICE
22/tcp   open  ssh
8080/tcp open  http-proxy
8443/tcp open  https-alt
9091/tcp open  xmltec-xmlmail

So, it seems to me that the issue may be that I have to make the aws package use port 8443 for Snowball Edge instead of 443 for the real S3. The code for connecting to S3 is pretty straight forward:

creds := credentials.NewStaticCredentials(s3Config.S3AccessKey, s3Config.S3SecretAccessKey, s3Config.S3Token)
_, err := creds.Get()

if err != nil {
    return nil, nil, err
}

if len(baseFolder) > 0 {
    baseFolder = baseFolder + "/"
}

cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds)
svc := s3.New(session.New(), cfg)

params := &s3.ListObjectsInput{
    Bucket:    aws.String(s3Config.S3BucketName),
    Prefix:    aws.String(baseFolder),
    Delimiter: aws.String("/"),
}

resp, err := svc.ListObjects(params)

So, the question is, how do I change the code to point to the Snowball Edge? I've tried mapping to the Snowball Edge in /etc/hosts from the Amazon S3 endpoint. I understand why this didn't work after discovering that the ports are different. I've played around with adding different forms of "WithEndpoint("...host...") with no success. Or, am I on the completely wrong track, and should be able to get the snowballAdapter to work with a Snowball Edge?

By the way, all the snowballEdge commands work as expected, so the device seems to be working fine, e.g.:

./snowballEdge list-access-keys

    {
      "AccessKeyIds" : [ "..." ]
    }

./snowballEdge get-secret-access-key --access-key-id ....

    [snowballEdge]
    aws_access_key_id = ...
    aws_secret_access_key = ...

And, I've used the correct keys associated with the device, and it does have the S3 Service configured in it:

./snowballEdge list-services
{
  "ServiceIds" : [ "s3" ]
}
  • 写回答

1条回答 默认 最新

  • duankezong4064 2018-05-08 18:59
    关注

    Snowball Edge is a very different beast than is AWS S3. In addition to the access key and secret access key, it requires that an endpoint and additional credentials in the form of a certificate. The real AWS S3 has a valid certificate, supported by the certificate authorities, but Snowball Edge has a self-signed certificate.

    The configuration can be created with this:

    cfg := aws.NewConfig().WithRegion(s3Config.S3Region).WithCredentials(creds).WithEndpoint(s3Config.S3Endpoint).WithHTTPClient(httpClient)
    svc = s3.New(session.New(), cfg)
    

    The endpoint looks like this (you see Snowball Edge responding on port 8443 in the question text above):

    https://192.168.1.4:8443
    

    The certificate is required to be formatted like you'd have in a file (including all the new line characters. It looks like this (again, the string you pass in has to include the new line characters after every line):

    -----BEGIN CERTIFICATE-----
    MIIC7zCCAdegAwIBAgIJBJZB/gkBP0B5MA0GCSqGSIb3DQEBCwUAMBYxFDASBgNV
    BAMMCzE3Mi4yMC4xLjE3NB4XDTE3MDQyODIwMTMwOFoXDTE5MDQxODIwMTMwOFow
    FjEUMBIGA1UEAwwLMTcyLjIwLjEuMTcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
    ggEKAoIBAQCrLWlTfrSj9R1of5Z98EHYIEEPBgnWxnlrvA+ryAzPmiXbYomI4Tpl
    PsuIA+7hGXG10H0zwlz0n22EUv4pE79toYcd3czOJUAHEuSelhtP7u91vM4GguFx
    A00gosu04RFUD+BYNeaLTQfd7vdmQB3bY3KEbn7Dfrs/1MYFhKb8J77mgCuUbAPu
    PNvwLoV+hBL+ndgs+bIu4MtXjUJDiigRZkacpMQaduDMqEq6seoc+JwrKNBjRBRu
    3l/fcQoWf+g902oZJaXXnVGqqb7o2YAQFehUAbmCJfuKFSl5tu0B+3KvQQni7lK+
    SV8WItdrPumS98BBlt6NpzgC5fTwCmapAgMBAAGjQDA+MAwGA1UdEwQFMAMBAf8w
    HQYDVR0OBBYEFMAvKzKgKI+izqPX6DJjJz/0fELtMA8GA1UdEQQIMAaHBKwUAREw
    DQYJKoZIhvcNAQELBQADggEBAGwyzmI+9psQu9/N/oClN7Lej7e4E8cC8vymVfPz
    fdW45IMNVEYHxHbu9+JzLOtLxfuDmD6B6fEYVoPubb6tsnacuwOSMZhTvhhy9nv2
    f+2Pslgj/kYTeMePbHOPTyQ4sd1BE7ALdNiL/hd08ZNhqObagixNYw9eYeHEStBy
    tOADKcY9gOxek1k9t+96nATgSy0WIytwra0uEgyipKQ2gXKpgg15SI4nDxQLLEgG
    lb3FtRk+PfJxQ4zbHZe/cRNflcGwVCefycLQOA2Sdr8pgHW7gvETu9i9ywF0UV6f
    b9wsPcDmg3EaxBa+wrLlYSzaPhI+rZYh6bpnTn311QIFZ+s=
    -----END CERTIFICATE-----
    

    Found the Snowball Edge to be pretty fast storage using S3. We were able to achieve about 1,100 MBytes/second of copy rate.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥15 R语言爬虫的时候元素和园代码不一样怎么解决呀
  • ¥15 VS2022多项目启动有问题
  • ¥15 SQL删除添加数据后序号不连续问题。
  • ¥15 首次运行OmniEvent运行报错
  • ¥15 有没有人知道这个问题怎么解决
  • ¥15 comsol电力电缆载流量仿真
  • ¥15 webSocket可以接TCP socket接口吗
  • ¥60 mpi并行出错,CFD++计算
  • ¥15 c#:vsto,powerpoint的外接程序中换主题颜色
  • ¥15 状态机/汽车转向灯/Sateflow