I have golang services with persistant Bigtable client. The services are making hundred of read/write operation on Bigtable per sec.
Every hours from the service boot, I experience hundred of errors like this one:
Retryable error: rpc error: code = Unavailable desc =
the connection is draining, retrying in 74.49241ms
The error are followed by an increased processing time I can't allow when thoses errors occur.
I was able to figure out that Bigtable client is using a pool of gRPC connections.
It seems that Bigtable gRPC server have a connection maxAge of 1 hour which can explain the error above and the processing time increasing during reconnection.
A maxAgeGrace configuration is supposed to give additional time to complete current operations and avoid all pool connections to terminate at the same time.
I increased connection pool size from default 4 to 12 with no real benefit
How do I prevent processing time to increase during reconnections and these error to happen, given my traffic will keep growing?