weixin_39715513
weixin_39715513
2020-12-01 23:15

Dynamically disable keep-alive / tear down a H2 connection using an ACL

What should haproxy do differently? Which functionality do you think we should add?

I'd like to be able to dynamically and cleanly tear down a client's TCP connection as part of a HTTP response, e.g.


http-response disconnect-afterwards if { res.hdr(x-client) disconnect }

What are you trying to do?

I'd like to force abusive clients to re-establish a new TCP connection and perform a new TLS handshake to slow them down.

Sending a connection: close or GOAWAY after sending the response to the current request should not affect legitimate traffic, apart from possibly introducing additional latency due to the reconnections. Specifically a legitimate user must not see requests failing like they would when I start sending a 429 too many requests. Injecting additional redirects as suggested in the mailing list thread does not play super nicely with POST requests, would require me to track additional state and is cheaply handled on the client side.

see: https://www.mail-archive.com/haproxy.org/msg38828.html

Output of haproxy -vv and uname -a


HA-Proxy version 2.3.1-1~bpo10+1 2020/11/15 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2022.
Known bugs: http://www.haproxy.org/bugs/bugs-2.3.1.html
Running on: Linux 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-2.3.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.3.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services :
    prometheus-exporter

Available filters :
    [SPOE] spoe
    [CACHE] cache
    [FCGI] fcgi-app
    [COMP] compression
    [TRACE] trace
</default></default></default>

该提问来源于开源项目:haproxy/haproxy

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

7条回答

  • weixin_39625586 weixin_39625586 5月前

    You already have "http-request reject" to forcefully close the H2 connection. However I'd advise you against doing that for what you're describing: forcing a client to perform another TLS handshake is going to slow you server down, because TLS handshakes are way more expensive for servers than clients. Maybe you have a specific use case in mind ?

    点赞 评论 复制链接分享
  • weixin_39715513 weixin_39715513 5月前

    You already have "http-request reject" to forcefully close the H2 connection.

    Oh, that's good to know, but it applies too early. I want to close the connection after successfully sending the response (i.e. effectively disable keep-alive).

    However I'd advise you against doing that for what you're describing: forcing a client to perform another TLS handshake is going to slow you server down, because TLS handshakes are way more expensive for servers than clients.

    Thanks for the warning. For this specific use case the odds are in my favor, though :smiley:

    Maybe you have a specific use case in mind ?

    Without going into too much detail: I want to protect the backend against a single (or small number of) clients tying up all the resources just by sending a large number of requests to a dynamically generated endpoint via keep-alive connections. The backend (for technical reasons) serves both dynamically generated responses as well as static files and legitimate browser clients sometimes tend to generate a large request rate (a single dynamically generated HTML file + all the images and JavaScript referenced there). For that reason I can't just send a 429 once the request rate exceeds some threshold, it would impact legitimate traffic (images or JavaScript might fail to load). I also can't determine in HAProxy whether a request will be dynamically generated or not, only the backend knows that.

    By forcing the client out of keep-alive when the request rate becomes too high a regular browser should notice nothing (especially once most of the files are cached), while these rogue clients will need to spend more resources by TCP handshaking and TLS handshaking. Additionally this allows other layers to detect this type of traffic, because new TCP connections are more visible than requests over a single keep-alive connection.

    点赞 评论 复制链接分享
  • weixin_39625586 weixin_39625586 5月前

    I see. Thanks for the context. During the development of the return action we discussed the possibility to implement the notion of a "class" of response, which would or would not produce a log, be counted as an error, or cause the connection to be closed. We later figure that most of these were already addressable using one more rule or by completing with a deny rule, even though for the long term it would be nice to have that. But I'm seeing that it wouldn't have been sufficient because in your case it's not a return but a rule that plugs on top of traffic.

    In your case, since you want to deliver the response correctly, it's more complicated, because no such code path exists to decide to send a GOAWAY frame after a valid response has been sent. And the only way we currently have to emit a GOAWAY frame is an error (which is what is done when using the reject rule). Maybe we should figure a way to send a GOAWAY frame with no error and make sure we don't keep track of it so that we're not tempted to abort too early the remaining processing. This would correspond to a "graceful connection closure" and could even be used to kill all idle H2 client connections on soft-stop or reload. But this requires some changes to the error tracking code so that we at least know what's the latest stream we've agreed to handle and ignore the other ones (because GOAWAY is a promise not to handle further streams so that client retry is safe).

    点赞 评论 复制链接分享
  • weixin_39715513 weixin_39715513 5月前

    Maybe we should figure a way to send a GOAWAY frame with no error

    This would be great, yes.

    even be used to kill all idle H2 client connections on soft-stop or reload

    I thought that a reload would already kill the H2 connections, because otherwise the old worker might never leave. Is that not the case?

    点赞 评论 复制链接分享
  • weixin_39625586 weixin_39625586 5月前

    On Fri, Nov 20, 2020 at 03:42:53AM -0800, Tim Düsterhus wrote:

    even be used to kill all idle H2 client connections on soft-stop or reload

    I thought that a reload would already kill the H2 connections, because otherwise the old worker might never leave. Is that not the case?

    I thought it was the case but failed to find it in the code. So I'm assuming it's closing on idle timeout.

    There are so many things to do at the same time :-(

    Willy

    点赞 评论 复制链接分享
  • weixin_39941859 weixin_39941859 5月前

    The documentation claims that for timeout client and timeout client-fin on H2 we send GOAWAY. Maybe that's a good starting point (assuming that this is true).

    But yeah it would be nice to also send it:

    • shutdown session X on the admin socket
    • soft-stop/reload
    • before hard-stop-after, not sure how exactly it works
    • and ... for the feature in question in this issue of course
    点赞 评论 复制链接分享
  • weixin_39625586 weixin_39625586 5月前

    Sure, on the moment we know we want to immediately close the connection, we attempt to send a GOAWAY (and timeout qualifies as this). Technically speaking the GOAWAY is currently sent as an error and sets the error condition on the connection, preventing it from further processing streams. This is why in the current state it's not compatible with the graceful close that Tim needs.

    I agree with the points you noted, except the hard-stop-after since this one is a violent termination to make sure we really quit. So there will not be any scheduling anymore to try to send anything.

    点赞 评论 复制链接分享

相关推荐