weixin_39625258
weixin_39625258
2020-11-30 09:15

2.0.6 cache with http/2 leads to wrong content delivered from cache

Output of haproxy -vv and uname -a


HA-Proxy version 2.0.6-2~bpo9+1 2019/09/18 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-2.0.6=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=40).
Built with OpenSSL version : OpenSSL 1.1.0k  28 May 2019
Running on OpenSSL version : OpenSSL 1.1.0k  28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.22 2016-07-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

Linux xxxxx 4.13.13-2-pve #1 SMP PVE 4.13.13-32 (Thu, 21 Dec 2017 09:02:14 +0100) x86_64 GNU/Linux
</default></default></default>

What's the configuration?


global
        log /dev/log        local0
        log /dev/log        local1 notice
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        maxconn 400000
        nbproc 1
        nbthread 12
        cpu-map auto:1/1-8 0-7
        daemon
        ssl-engine rdrand
        ssl-mode-async
        ca-base /etc/ssl/certs
        crt-base /etc/haproxy/ssl
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
        tune.ssl.default-dh-param 2048
        tune.ssl.cachesize 10000000
        tune.ssl.lifetime 600
        stats socket /run/haproxy/admin.sock mode 660 level admin
        lua-load /etc/haproxy/delay.lua

defaults
        log        global
        mode        http
        option        httplog
        option        dontlognull
        option        http-server-close
        option        redispatch
        option        persist
        option        abortonclose
        timeout        connect 16s
        timeout client 60s
        timeout server 300s
        timeout check 30s
        timeout http-keep-alive 4s
        timeout tunnel 1h
        timeout tarpit 8s
        default-server inter 8s
        maxconn 400000
        backlog 20000
        balance leastconn
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

cache content
        total-max-size 1024
        max-age 600
        max-object-size 10485760

cache content_web6
        total-max-size 512
        max-age 600
        max-object-size 104857600

listen pve
        bind x.x.x.x:80
        bind x.x.x.x:80
        bind x.x.x.x:443 ssl crt all/
        bind x.x.x.x:443 ssl crt all/
        use_backend bk_letsencrypt if { path_beg /.well-known/acme-challenge/ }
        use_backend bk_local if { path_beg /vvvvvvv }
        use_backend bk_stats if { path_beg /uuuuuuu }

#### STICK TABLES ####
backend st_src_global
        stick-table type ip size 1m expire 60s store conn_cur,conn_rate(30s),http_req_rate(30s)
backend st_src_iframe
        stick-table type ip size 100k expire 10s store http_req_rate(10s)
backend st_src_ajax
        stick-table type ip size 1m expire 60s store http_req_rate(60s)
backend st_src_kkkkk_ajax
        stick-table type ip size 1m expire 60s store http_req_rate(60s)
backend st_src_404
        stick-table type ip size 1m expire 60s store http_req_rate(60s)
backend st_abuse
        stick-table type ip size 100k expire 60m store gpc0
#### TARPIT ####
backend tarpit_abuse
        http-request tarpit
backend tarpit_404
        http-request tarpit
backend tarpit_ajax
        http-request tarpit
backend tarpit_iframe
        http-request tarpit

#### VIP1 ####
frontend vip-main
        bind vip1:80
        bind vip1:443 ssl crt all/ alpn h2,http/1.1
        maxconn 200000
        option http-keep-alive
        capture request header Host len 32
        ## ACL
        acl url_letsencrypt path_beg /.well-known/acme-challenge/
        acl is_cf src -f /etc/haproxy/cf-ips-v4
        acl hdr_cf_cip req.hdr(cf-connecting-ip) -m found
        acl wl src -f /etc/haproxy/ip_whitelist.lst
        acl acl_xxx src x.x.x.x/24 x.x.x.x/27 x.x.x.x/24
        acl ajax_url path_beg /ajax.php
        acl xpromo urlp(class) -i crosspromo
        acl abuse_ua hdr_sub(user-agent) -f /etc/haproxy/abuse-ua.lst
        acl has_ref hdr_cnt(referer) gt 0
        acl has_ua hdr_cnt(user-agent) gt 0
        acl acl_no_ssl hdr(host) -i ###some domains### 
        acl bbb_priv_hosts hdr(Host) -i ###some domaine###
        ## tcp-request
        tcp-request inspect-delay 10s
        tcp-request connection accept if wl OR is_cf
        tcp-request connection track-sc0 src table st_src_global
        tcp-request connection reject if { sc0_conn_cur(st_src_global) gt 20 }
        ## http-request
        http-request set-header X-Forwarded-Proto https if { ssl_fc } !is_cf
        http-request set-src req.hdr(CF-Connecting-IP) if is_cf hdr_cf_cip
        http-request set-header X-Forwarded-For %[src]
        http-request redirect scheme https code 301 if !{ ssl_fc } !acl_no_ssl
        http-request deny if bbb_priv_hosts !acl_xxx !url_letsencrypt
        http-request allow if wl OR is_cf
        http-request track-sc1 src table st_src_iframe if { path_beg /yyy/iframe.php }
        http-request track-sc1 src table st_src_ajax if { path_beg /ajax.php }
        http-request track-sc1 src table st_src_kkkkk_ajax if { path_beg /kkkkk-ajax.php }
        http-request tarpit if ajax_url !xpromo !has_ref
        http-request tarpit if ajax_url !xpromo !has_ua
        http-request tarpit if ajax_url !xpromo abuse_ua
        http-request lua.add_delay if { sc0_conn_rate(st_src_global) gt 600 }
        http-request lua.add_delay if { sc0_http_req_rate(st_src_global) gt 1200 }
        ## http-response
        http-response allow if wl OR is_cf
        http-response track-sc1 src table st_src_404 if { status 404 }
        ## redirect
        ## redirect scheme https code 301 if !{ ssl_fc }
        ## Stats
        use_backend bk_stats if { path_beg /uuuuuuu }
        ## Letsencrypt
        use_backend bk_letsencrypt if { path_beg /.well-known/acme-challenge/ }
        ## Tarpit
        use_backend tarpit_abuse if { src_get_gpc0(st_abuse) ge 1 }
        use_backend tarpit_404 if { src_http_req_rate(st_src_404) gt 120 } { src_inc_gpc0(st_abuse) ge 0 }
        use_backend tarpit_ajax if { src_http_req_rate(st_src_ajax) gt 900 } { src_inc_gpc0(st_abuse) ge 0 }
        use_backend tarpit_ajax if { src_http_req_rate(st_src_kkkkk_ajax) gt 30 } { src_inc_gpc0(st_abuse) ge 0 }
        use_backend tarpit_iframe if { src_http_req_rate(st_src_iframe) gt 10 } { src_inc_gpc0(st_abuse) ge 0 }
        ## Clicker
        use_backend bk_test_zzz_com_clicker if { hdr(host) -i test.zzz.com } { path_beg /auth/ /clicker/ }
        use_backend bk_test_zzz_com_content if { hdr(host) -i test-content.zzz.com } { path_beg /content/ }
        use_backend bk_zzz_com_clicker      if { hdr(host) -i www.zzz.com } { path_beg /auth/ /clicker/ }
        use_backend bk_zzz_com_content      if { hdr(host) -i content.zzz.com } { path_beg /content/ }
        use_backend bk_aaa_zzz_com_clicker      if { hdr(host) -i aaa.zzz.com } { path_beg /auth/ /clicker/ }
        use_backend bk_aaa_zzz_com_content      if { hdr(host) -i aaa.zzz.com } { path_beg /content/ }
        use_backend bk_yyy_zzz_com_clicker if { hdr(host) -i yyy.zzz.com } { path_beg /auth/ /clicker/ }
        use_backend bk_yyy_zzz_com_content if { hdr(host) -i yyy.zzz.com } { path_beg /content/ }
        ## Chat
        acl acl_mmmm path_beg /mmmm/ /socket.io
        use_backend bk_mmmm_www_domain_com if acl_mmmm { hdr(Host) -i www.domain.com dev2.domain.com }
        use_backend bk_mmmm_yyy_llll_com if acl_mmmm { hdr(Host) -i yyy.llll.com dev2.llll.com }
        use_backend bk_mmmm_www_jjjjjj_com if acl_mmmm { hdr(Host) -i www.jjjjjj.com dev2.jjjjjj.com }
        use_backend bk_mmmm_lllll_domain_com if acl_mmmm { hdr(Host) -i lllll.domain.com }
        use_backend bk_mmmm_kkkkk_domain_com if acl_mmmm { hdr(Host) -i kkkkk.domain.com }
        use_backend bk_mmmm_eeeee_domain_com if acl_mmmm { hdr(Host) -i eeeee.domain.com }
        use_backend bk_mmmm_yyy_jjjjjj_com if acl_mmmm { hdr(Host) -i yyy.jjjjjj.com }
        use_backend bk_mmmm_dev if acl_mmmm { hdr_reg(host) -i ^dev2\. ^sbox-yyy\. ^lllll-dev2\. ^test\. }
        use_backend bk_mmmm_prod if acl_mmmm
        ## 
        use_backend bk_bbb_xxx_int if { hdr(Host) -i bbb-int.xxx.com bbb.xxx.com bbb-nnnn.xxx.com }
        use_backend bk_bbb_xxx_ext if { hdr(Host) -i bbb-ext.xxx.com }
        use_backend bk_bbb_xxx_int_dev if { hdr(Host) -i bbb-int-dev.xxx.com }
        use_backend bk_bbb_xxx_ext_dev if { hdr(Host) -i bbb-ext-dev.xxx.com }
        ## Content --> nginx
        use_backend bk_content_web6 if { hdr(Host) -i content.zzz.com }
        use_backend bk_content if { hdr_reg(host) -i ^content\. }
        ## eees
        acl acl_eees hdr(host) -i ### some_domains ###
        use_backend bk_eees if acl_eees
        ## WEB6 specific
        acl acl_web6 hdr_reg(host) -i zzz\.com$
        use_backend bk_web6 if acl_web6
        ## --> web1/2
        acl acl_yyy1 hdr_reg(host) -i ### some_domains ###
        acl acl_yyy2 hdr_reg(host) -i ### some_domains ###
        use_backend bk_web1 if acl_yyy1
        use_backend bk_web2 if acl_yyy2
        ## DEFAULT --> web5
        default_backend bk_web5

frontend vip-sssss
        bind 94.75.250.121:80
        bind 94.75.250.121:443 ssl crt all/
        maxconn 40000
        capture request header Host len 32
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        redirect scheme https if { hdr(Host) -i vip2.g.g.g } !{ ssl_fc }
        option forwardfor header X-Forwarded-For
        use_backend bk_stats if { path_beg /uuuuuuu }
        use_backend bk_letsencrypt if { path_beg /.well-known/acme-challenge/ }
        default_backend bk_wpccc

#### BACKENDS ####
backend bk_letsencrypt
        option httpclose
        server letsencrypt 127.0.0.1:54321

backend bk_local
        option httpclose
        server local 127.0.0.1:8000

backend bk_stats
        stats uri /haproxy?###
        stats enable
        stats auth  admin:###
        stats admin if TRUE

## CONTENT
backend bk_content
        option http-keep-alive
        http-reuse always
        http-request del-header Cookie
        http-request cache-use content
        http-response cache-store content
        server web1-nginx y.y.y.21:8080 maxconn 8000 check backup
        server web2-nginx y.y.y.22:8080 maxconn 8000 check backup
        server web5-nginx y.y.y.7:8080 maxconn 8000 check
        server web6-nginx y.y.y.8:8080 maxconn 8000 check backup

backend bk_content_web6
        option http-keep-alive
        http-reuse always
        http-request del-header Cookie
        http-request cache-use content_web6
        http-response cache-store content_web6
        server web5-nginx y.y.y.7:8080 maxconn 8000 check backup
        server web6-nginx y.y.y.8:8080 maxconn 8000 check

## WEB1
backend bk_web1
        option http-keep-alive
        option persist
        http-reuse always
        cookie HAPBK insert indirect nocache
        server web1 y.y.y.21:8000 maxconn 1600 check cookie web1

## WEB2
backend bk_web2
        option http-keep-alive
        option persist
        http-reuse always
        cookie HAPBK insert indirect nocache
        server web2 y.y.y.22:8000 maxconn 1600 check cookie web2

## WEB1/WEB2
backend bk_web1_web2
        option http-keep-alive
        option persist
        http-reuse always
        cookie HAPBK insert indirect nocache
        server web1 y.y.y.21:8000 maxconn 1600 check cookie web1
        server web2 y.y.y.22:8000 maxconn 1600 check cookie web2 backup


## WEB5
backend bk_web5
        option http-keep-alive
        no option persist
        http-reuse always
        cookie HAPBK insert indirect nocache
        server web5 y.y.y.7:8000 maxconn 3000 check cookie web5

## WEB6
backend bk_web6
        option http-keep-alive
        option persist
        http-reuse always
        cookie HAPBK insert indirect nocache
        server web6 y.y.y.8:8000 maxconn 3000 check cookie web6

## Landings 
backend bk_eees
        option http-keep-alive
        option persist
        http-reuse always
        server fff1 w.w.w.w2:8080 maxconn 200 check
        server fff2 w.w.w.w3:8080 maxconn 200 check backup

## WEB6-NODE
backend bk_mmmm_prod
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3000 maxconn 20000
backend bk_mmmm_dev
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-dev y.y.y.8:6000 maxconn 20000
backend bk_mmmm_www_domain_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3010 maxconn 20000
backend bk_mmmm_yyy_llll_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3020 maxconn 20000
backend bk_mmmm_www_jjjjjj_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3030 maxconn 20000
backend bk_mmmm_lllll_domain_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3040 maxconn 20000
backend bk_mmmm_kkkkk_domain_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3050 maxconn 20000
backend bk_mmmm_eeeee_domain_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3060 maxconn 20000
backend bk_mmmm_yyy_jjjjjj_com
        reqirep ^([^\ :]*)\ /mmmm/(.*) \1\ /\2
        rspirep ^Location:\ (https?://[^/]*)?/(.*)$ Location:\ \1/\2
        server node-prod y.y.y.8:3070 maxconn 20000
backend bk_test_zzz_com_clicker
        server node-prod y.y.y.8:7001 maxconn 20000
backend bk_test_zzz_com_content
        server node-prod y.y.y.8:7011 maxconn 20000
backend bk_zzz_com_clicker
        server node-prod y.y.y.8:7200 maxconn 20000
backend bk_zzz_com_content
        server node-prod y.y.y.8:7210 maxconn 20000
backend bk_aaa_zzz_com_clicker
        server node-prod y.y.y.8:7300 maxconn 20000
backend bk_aaa_zzz_com_content
        server node-prod y.y.y.8:7310 maxconn 20000
backend bk_yyy_zzz_com_clicker
        server node-prod y.y.y.8:7400 maxconn 20000
backend bk_yyy_zzz_com_content
        server node-prod y.y.y.8:7410 maxconn 20000
##  backends
backend bk_bbb_xxx_int
        server node-prod y.y.y.8:5000 maxconn 20000
backend bk_bbb_xxx_ext
        server node-prod y.y.y.8:5005 maxconn 20000
backend bk_bbb_xxx_int_dev
        server node-prod y.y.y.8:5990 maxconn 20000
backend bk_bbb_xxx_ext_dev
        server node-prod y.y.y.8:5995 maxconn 20000

## WP
backend bk_wpccc
        option http-server-close
        server web3 y.y.y.51:80 maxconn 400 check
        server web4 y.y.y.52:80 maxconn 400 check backup

Steps to reproduce the behavior

Enable HTTP/2 and caching at the same time (alpn h2,http/1.1)

Load a page with several images from content domain in any browser supporting http/2. Tested with several browsers and latest chrome version.

Path is : frontend vip-main use_backend bk_content if { hdr_reg(host) -i ^content. }

Actual behavior

Loaded page contains lots of images that are random and do not correspond to the requested URI. Seems that haproxy sends a random content from cache.

Expected behavior

Content corresponding to request URL shoube be the one expected, not random.

Do you have any idea what may have caused this?

Probably cache corruption with http/2. Don't know if the objects are correct in cache or if it is mixed after, when fetched and multiplexed in http/2.

Do you have an idea how to solve the issue?

We disabled http/2, reloaded, and it is ok. Correct content is served from cache with http/1.1.

However we did not test with http/2 enabled and cache disabled, it is planned for next tuesday because of no modification allowed on weekend.

该提问来源于开源项目:haproxy/haproxy

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

5条回答

  • weixin_39940788 weixin_39940788 5月前

    Can you see #290 and try some of the options suggested by in https://github.com/haproxy/haproxy/issues/290#issuecomment-533464852 .

    点赞 评论 复制链接分享
  • weixin_39952190 weixin_39952190 5月前

    My last fix on the H2 multiplexer is a good candidate to fix this bug. Could you make some tests with this patch ? It was backported to 2.0.

    点赞 评论 复制链接分享
  • weixin_39625258 weixin_39625258 5月前

    Unfortunately I cannot reproduce without significative load and I can't test a patch in production. All I can tell is that with http/2 disabled it is working as expected on 2.0.6.

    点赞 评论 复制链接分享
  • weixin_39952190 weixin_39952190 5月前

    FYI, the release 2.0.7 was published.

    点赞 评论 复制链接分享
  • weixin_39952190 weixin_39952190 5月前

    Don't know if it was related to the mentioned fixed (6884aa3) and no feedback since a while. I'm closing this issue tagging it as "cannot reproduce". The issue #290 is still opened. If there is a bug here, it is probably the same.

    点赞 评论 复制链接分享

相关推荐