weixin_39896617
weixin_39896617
2020-12-27 08:58

realloc error

Hi On a fresh Ubuntu 14.04.5 LTS I install via apt: - build-essential - libboost-dev - libboost-all-dev - gperf - libevent-dev - uuid-dev - libssl-dev - libmysqlclient-dev

I compile gearmand with:


wget https://github.com/gearman/gearmand/releases/download/1.1.18/gearmand-1.1.18.tar.gz
./configure --enable-ssl
make
make install

I run gearmand with on boot with:


$ cat /etc/init/gearman-job-server-gearmand.conf 
# Gearman worker gearmand

description gearmand

start on started mysql-gearman
stop on runlevel [!2345]

pre-start script
    mkdir -p /var/log/gearmand
    chown -R gearman:gearman /var/log/gearmand
    mkdir -p /var/run/gearmand
    chown -R gearman:gearman /var/run/gearmand
end script

kill timeout 30

respawn

console log

exec sudo -u gearman /usr/local/sbin/gearmand --listen=127.0.0.1 --port=PORT --queue-type=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=USER --mysql-password=PASSWORD --mysql-db=DATABASE_NAME --mysql-table=DATABASE_TABLE --pid-file=/var/run/gearmand/gearmand.pid --round-robin --threads=0 --log-file=/var/log/gearmand/gearmand.log

I see in /var/log/gearmand/gearmand.log about 22M rows per day like this:


  ERROR 2018-05-24 06:27:51.000000 [  main ] realloc(char, count: 18446744073709551615 size: 1) -> libgearman-server/io.cc:823

I would like to ask if you have any insights for the severity of the error or how to provide more information about it?

该提问来源于开源项目:gearman/gearmand

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

6条回答

  • weixin_39634052 weixin_39634052 3月前

    So this is the code you're hitting:

        
        packet->data= static_cast<char>(realloc(NULL, packet->data_size));
        if (not packet->data)
        {
          // Server up the memory error first, in case _connection_close()
          // creates any.
          gearmand_merror("realloc", char, packet->data_size);
          _connection_close(connection);
          return GEARMAND_MEMORY_ALLOCATION_FAILURE;
        }
    </char>

    Earlier in the code, there's a guard for 0-length data, so according to the man page of realloc, a NULL return means memory allocation failed. My guess is that you have very large job payloads, and sometimes your server cannot allocate memory. If I had to guess why I'd say it's due to ulimits. Perhaps raise your ulimit to allow more memory to be allocated at once. Whenever this is happening, gearmand is closing the connection with the client.

    So, to be clear, yes this is a real error. There's not really anything I think we can do in gearmand to avoid this, but please do feel free to reopen this if you feel that there's more to consider.

    点赞 评论 复制链接分享
  • weixin_39566578 weixin_39566578 3月前

    https://github.com/p-alik/perl-Gearman/issues/44 describes how the issue could be reproduced.

    点赞 评论 复制链接分享
  • weixin_39886547 weixin_39886547 3月前

    p-alik/perl-Gearman#44 describes how the issue could be reproduced.

    At least in this case, it doesn’t seem like the job payload is as large as gearmand thinks it is. Do you agree, -alik? Could packet->data_size contain garbage somehow?

    点赞 评论 复制链接分享
  • weixin_39566578 weixin_39566578 3月前

    Actually the test script doesn't send any payload at all.

    点赞 评论 复制链接分享
  • weixin_39886547 weixin_39886547 3月前

    Actually the test script doesn't send any payload at all.

    Maybe that's the problem. Perhaps packet->data_size isn't being set correctly when packet->data is a zero-length string?

    点赞 评论 复制链接分享
  • weixin_39566578 weixin_39566578 3月前

    Maybe that's the problem. Perhaps packet->data_size isn't being set correctly when packet->data is a zero-length string?

    It looks like gearmand logs realloc-Error, if client sends OPTION_REQ-Request for exceptions and worker doesn't send WORK_COMPLETE-Request.

    To reproduce gearmand's behaviour with 16-background.t you have to remove https://github.com/p-alik/perl-Gearman/commit/e085ee624526fb620eff425e1263ed59fcb63c32#diff-16d3e60dc6c1b4c9c0f396518acde50aR19 and https://github.com/p-alik/perl-Gearman/commit/4b1a8520aec9cb0a1cccf44404b44c6447e0eeb7

    点赞 评论 复制链接分享

为你推荐