weixin_39788386
weixin_39788386
2020-11-25 09:27

Performance issue with unusually high "max open files" ulimit

Hello,

don't ask me why (or where it's configured) but our Jenkins server is configured for 1M max open files, as per /proc//limits.

I see very poor runtime of tests, multiple orders of magnitudes longer runtimes for some tests, compored to running our tests locally on my workstation (from <1s to >30s)

I found that the open files ulimit is the main issue. When I wrap the tests with a script that does "ulimit -n 256", the runtime of the affected tests goes back to normal. I found this in strace traces, where I saw that fcntl() is called for every possible file descriptor. The code is in BoxFort, specifically this loop: https://github.com/Snaipe/BoxFort/blob/master/src/sandbox-posix.c#L508

What's curious: some tests do not show the poor performance. Even if I artificially compile the exact same code in two tests, one is quick even inside Jenkins but the other one slow as hell.

Do you think the loop can be smarter to handle such systems?

该提问来源于开源项目:Snaipe/Criterion

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

5条回答

  • weixin_39788386 weixin_39788386 5月前

    I found why there is 1M max open files. It's the default behavior of the openjdk jre that runs Jenkins to ramp up the soft fd limit to the hard one (so 1K -> 1M max open files).

    I can pass -XX:-MaxFdLimit to mitigate but I still think this could be improved in Criterion.

    点赞 评论 复制链接分享
  • weixin_39760368 weixin_39760368 5月前

    You're absolutely right -- BoxFort is a bit dumb right now on the account that this was the most portable way to approach this, although very inefficient.

    I can add a special case on linux, and essentially go through all entries under /proc//fd.

    点赞 评论 复制链接分享
  • weixin_39760368 weixin_39760368 5月前

    I looked at this today -- it's weird, because boxfort should already be looking at opened file descriptors only. Does your setup have /dev/fd? My laptop seems to make that a symlink to /proc/self/fd, and it's the file BoxFort is looking for.

    点赞 评论 复制链接分享
  • weixin_39788386 weixin_39788386 5月前

    The tests run in LXC containers. Maybe they get a fresh /dev. I cannot double check right now

    点赞 评论 复制链接分享
  • weixin_39760368 weixin_39760368 5月前

    Here's a patch that falls back to /proc/self/fd in case /dev/fd isn't there: http://ix.io/2gJx

    点赞 评论 复制链接分享

相关推荐