AWS / ALB，http / 2和GOAWAY

Question

We recently switched from ELB to ELB2/ALBs and occasionally our go http/2 clients are seeing GOAWAY messages from our Application Load Balancers which I'm unable to explain. The target group servers only support http/1.1 and our load balancers should always have at least one healthy server in rotation.

I can reliably reproduce the GOAWAY when registering a new instance in the ALB. The ALB returns GOAWAY when the target is in the "Initial" state. Further, even though the ALB responds with GOAWAY the request successfully makes it to the other instance registered in the target group. So, given instances web0 and web1, if I deregister web0 and re-register that target I can reliably reproduce the GOAWAY if I make a request while web0 is in "Initial". However our logs show that web1 successfully handled the request.

Our client is a Go program using http.DefaultClient. I can reproduce this behavior using both Go 1.7 and 1.8beta2.

When this occurs our client logs more details about the HTTP/2 response:

http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=NO_ERROR, debug=""

I'd like to better understand what's going on here. Should either the go http2 package or our code automatically handle the GOAWAY by retrying the request? I'm not familiar enough with http2 to know if GOAWAY is expected, which implies that our Go client shouldn't handle it as an error condition, or if this indicates that something is going wrong at the ALB.

Answer 1

About GOAWAY

The GOAWAY frame carries three pieces of information that could help you troubleshooting the issue:

 +-+-------------------------------------------------------------+
 |R|                  Last-Stream-ID (31)                        |
 +-+-------------------------------------------------------------+
 |                      Error Code (32)                          |
 +---------------------------------------------------------------+
 |                  Additional Debug Data (*)                    |
 +---------------------------------------------------------------+

• Last-stream-ID is the last ID that was correctly processed. This might help understand what's going on: the RFC has some advice on how to achieve a graceful shutdown: First send a GOAWAY frame with Last-Stream-ID with NO_ERROR, to let the client know that a shutdown is about to come, then after some time, send another GOAWAY frame with Last-Stream-ID set to the actually last processed ID. So that the client knows what's been passed on. Here's the relevant extract, from RFC7540, 6.8 GOAWAY

A server that is attempting to gracefully shut down a connection SHOULD send an initial GOAWAY frame with the last stream identifier set to 2^31-1 and a NO_ERROR code. This signals to the client that a shutdown is imminent and that initiating further requests is prohibited. After allowing time for any in-flight stream creation (at least one round-trip time), the server can send another GOAWAY frame with an updated last stream identifier. This ensures that a connection can be cleanly shut down without losing requests.

• The error code and the additional debug data (a string), will contain additional information that explain what's going on. RFC 7540, 7. Error Codes has the list of possible error codes. Then depending on the server implementation you might have a string narrowing down the error. For example in H2O, the server sends found an upper-case letter in header name when an upper case letter was found in a header name.

This particular GOAWAY

http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=NO_ERROR, debug=""

Since the server is sending NO_ERROR, your client should simply try to reconnect, and not treat the message as an error.

As to why the ALB is sending GOAWAYs... I'm not sure, can you give us more details about that?