间歇性错误getsockopt：Http帖子上的连接被拒绝错误

Question

There are two go apps, one is stapi listening on port 8050 and providing RESTful APIs, another is client to consume those APIs.

Both are running on different servers, client is often getting error when calling APIs with HTTP POST method. Below are few lines from client log (real IP replaced with imaginary one)

2018/02/17 11:42:58 ERROR: [DoLogin] API Error: [Post https://123.123.123.123:8050/v1/st/verifyuser: dial tcp 123.123.123.123:8050: getsockopt: connection refused]
2018/02/17 11:47:14 ERROR: [CreateAttempt] Error: [Post https://123.123.123.123:8050/v1/userattempts/createattempt: dial tcp 123.123.123.123:8050: getsockopt: connection refused]

It is intermittent and making the app unreliable, out of approx 1k request i got such error for approx 50+ request.

Initially stapi was listening on all IPs

httpSrv := http.Server{
    Addr:         ":8050",
    Handler:      router, // < gin router
    ...
}

But after reading the workaroung in Golang HTTP Post error: connection refused i modified the stapi app and make it listening on different IPs, as shown below

$ sudo lsof -i -P -n | grep LISTEN
stapi     4775  samtech   10u  IPv4 2388179      0t0  TCP 123.123.123.123:8050 (LISTEN)
stapi     4775  samtech   11u  IPv6 2388181      0t0  TCP [::1]:8050 (LISTEN)
stapi     4775  samtech   12u  IPv4 2388183      0t0  TCP 127.0.0.1:8050 (LISTEN)

But still the issue is same, what else i should check and fix ? Please suggest.

API is protected with JWT, here is how client is making POST requests

func (w *OST) DoLogin(c *gin.Context) {
    ...
    ud := stapimodels.UserLogin{}
    err := c.BindJSON(&ud)
    ...
    //call api to save user response
    url := config.AppConfig.APIBaseURL + "st/verifyuser"
    res, err := api.JwtApi.APIPost(url, &ud)
    if err != nil {
            g.Logger.Errorm("DoLogin", "Error: %v", err)
            t.Error("Error", err.Error())
            return
    }
    ...
}

//APIPost - call given apiurl with POST method and pass data
func (j *JwtAPI) APIPost(apiurl string, postdata interface{}) (*APIResult, error) {
        if postdata == nil {
                return nil, fmt.Errorf("postdata is nil")
        }
        jsondata, err := toJSON(postdata)
        if err != nil {
                return nil, err
        }

        resp, err := j.makeRequest(http.MethodPost, apiurl, jsondata)
        if err != nil {
                return nil, err
        }

        defer resp.Body.Close()
        res := APIResult{}
        json.NewDecoder(resp.Body).Decode(&res)
        return &res, nil
}

//makeRequest makes http request for given url with given method
// also inject Authorization Header
func (j *JwtAPI) makeRequest(method, apiurl string, body io.Reader) (*http.Response, error) {
        retry := 0
        //Create []byte buffer from body - so it can be passed in further retries
        var buf []byte
        if body != nil {
                buf, _ = ioutil.ReadAll(body)
        }

        r, err := http.NewRequest(method, apiurl, bytes.NewReader(buf))
        if err != nil {
                return nil, err
        }
        r.Header.Set("Authorization", "bearer "+j.token.AccessToken)
        r.Header.Set("Content-Type", "application/json")

        client := j.getClient()
        resp, err := client.Do(r)
        if err != nil {
                return nil, err
        }
        return resp, nil
}

func (j *JwtAPI) getClient() *http.Client {
        // default timeout (if not set by client)
        timeoutInSec := 10
        if j.Timeout.Seconds() > 0 {
                // client sets timeout, so use it
                timeoutInSec = int(j.Timeout.Seconds())
        }

        client := &http.Client{
                Timeout: time.Second * time.Duration(timeoutInSec),
        }
        return client
}

Answer 1

To make your code more resilient you should add some retries with back-offs, so even when the connection was refused it is still working.

Connection refused means that the port is not opened. Is there any firewall or proxies in between? The authentication part shouldn't matter here because it doesn't even get to this point.

Some things that you can check:

• Make sure the service is running
• Check for firewall configuration
• Implement retries for resilience
• Is the IP-Address fixed? Is Dynamic DNS used and maybe not updated?

Package for back-off retrying

As for implementing the back-off you might try this package:

https://github.com/cenkalti/backoff

It is listing examples on how to use it and it's pretty much exactly what you need:

// An operation that may fail.
operation := func() error {
    // do the request here and check the response code (or check the response body depending on your need) . e.g. above 500 should retry, above 400 and below 500, it should be a client side error and retrying might not help much
    return nil // or an error
}

err := Retry(operation, NewExponentialBackOff())
if err != nil {
    // Handle error.
    return
}

// Operation is successful.