Golang在Ubuntu 14.04 LTS中大量读取tcp ip：port I / O超时

Question

I wrote a golang program which run well in the past several months in ubuntu 12.04 LTS until I upgraded it to 14.04 LTS

My program is focused on sending HTTP requests which send about 2-10 HTTP requests per second. The HTTP request address vary.

When the problem occurs, first, some of the requests shows read tcp [ip]:[port]: i/o timeout, then after several minutes all requests show read tcp [ip]:[port]: i/o timeout, not any request can be sent.

I restart the program, everything become right again.

All of our servers(2 server) have such problem after upgraded from 12.04 to 14.04

I create new goroutine for every request

the problem does not occur in the same interval, sometimes it won't occur one or two day, sometimes It occur twice in an hour

Bellow is my code requesting HTTP Address:

t := &http.Transport{
    Dial:            timeoutDial(data.Timeout),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
//req := s.ParseReq(data)
req := data.convert2Request()
if req == nil {
    return
}

var resp *http.Response
if data.Redirect {
    c := &http.Client{
        Transport: t,
    }
    resp, err = c.Do(req)
} else {
    resp, err = t.RoundTrip(req)
}

data.updateTry()

r := s.ParseResp(data, resp, err)

updateTry:

func (d *SendData) updateTry() {
    d.Try++
    d.LastSend = time.Now()
}

timeoutDial:

func timeoutDial(timeout int) func(netw, addr string) (net.Conn, error) {
    if timeout <= 0 {
        timeout = 10
    }
    return func(netw, addr string) (net.Conn, error) {
        deadline := time.Now().Add(time.Duration(timeout) * time.Second)
        c, err := net.DialTimeout(netw, addr, time.Second*time.Duration(timeout+5))
        if err != nil {
            return nil, err
        }
        c.SetDeadline(deadline)
        return c, nil
    }
}

and My dealing with response is:

func (s *Sender) ParseResp(data SendData, resp *http.Response, err error) (r Resp) {
    r = Resp{URL: data.URL}
    if err != nil {
        r.Err = err.Error()
    } else {
        r.HttpCode = resp.StatusCode
        r.Header = resp.Header
        r.URL = resp.Request.URL.String()
        defer resp.Body.Close()
        // we just read part of response and log it.
        reader := bufio.NewReader(resp.Body)
        buf := make([]byte, bytes.MinRead) // 512 byte
        for len(r.Body) < 1024 {           // max 1k
            var n int
            if n, _ = reader.Read(buf); n == 0 {
                break
            }
            r.Body += string(buf[:n])
        }
    }
    return
}

I also found setting in /etc/sysctl.conf which can make the problem happen less frequently:

net.core.somaxconn = 65535
net.netfilter.nf_conntrack_max = 655350
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

I need help for solving this problem.

It seems like this but I don't see any solution https://bugs.launchpad.net/juju-core/+bug/1307434

Answer 1

Without seeing the code to timeoutDial, my wild guess is that you don't close the connection when you're done with it.