I have a website running with around 7 servers (C#). And there is a gRPC service (golang) running with 3 instances. Each web server connects to and makes calls to the gRPC service. There are around 8000 calls per minute to the service.
The call to this service is not that critical, so lately we reduced the deadline of the call to 20 milliseconds. Here we noticed something strange. There was a spike in the "deadline exceeded" errors every hour throughout the day. And it happens exactly at 0th minute i.e. 2pm 3pm 4pm etc.
Why does this happen?
I came across this link saying gRPC resets the connection every hour, but nothing more than that.
So my question is does gRPC internally refresh the connection every hour. If yes is there anyway to tweak this behavior. If no then can someone give some direction as to how I can debug why this is happening.