I have a statsd client that pushes to a kubernetes-hosted statsd service in tcp.
When I redeploy the service and update its pods on the server, the data stops flowing but I have no error on the statsd client.
Based on this thread, I have added a read detection to my golang library to see if I receive a message, but nothing.
I start the connection, send statsd packets from the client, then after I while, I scale the statsd Service down to 0 pods, but the client continues to successfully read and write from the tcp channel without errors!
When I scale the deployment back to 1, the old client does not reconnect and the client and the server continue to remain non-communicating forever.
How can I receive an error from the client when the server replicas go to 0 in the Kubernetes service?
The statsd client is a fork of https://github.com/alexcesaro/statsd
Note that we are aware that statsd is used mostly in UDP but this is a custom flavor and for several reason we need one instance of this in TCP (already have several UDP ones and they don't have this problem)
...
c.w, err = net.DialTimeout(c.network, c.addr, 5*time.Second)
...
func (c *conn) checkTCPConnectionOpen() error {
if c.network[:3] == "tcp" {
one := []byte{}
_, err := c.w.Read(one)
return err
}
return nil
}
func (c *conn) flush(n int) {
if len(c.buf) == 0 {
return
}
if n == 0 {
n = len(c.buf)
}
err := c.checkTCPConnectionOpen()
c.handleError(err)
_, err = c.w.Write(c.buf[:n])
c.handleError(err)
if n < len(c.buf) {
copy(c.buf, c.buf[n:])
}
c.buf = c.buf[:len(c.buf)-n]
}
