weixin_39768247
weixin_39768247
2021-01-03 04:46

add proper sleep to upstart system job.

Problem

When upstart system job is failed, upstart supervisor will run again soon. Sometime this it's too fast. Too fast restarting will raech to respawn limitation soon.

Solution

Add proper sleep to before exec not to reach respawn limitation soon.

Btw. LB instance already uses same sleep.

https://github.com/axsh/wakame-vdc/blob/develop/vmapp/load_balancer/etc/init/haproxy_updater.conf#L12-L16:


  # Sometimes this job failed before initializing networking with wakame-init.
  #
  # If this job is failed, upstart supervisor will run again soon. It's too fast.
  # Therefore a proper sleep is needed.
  sleep ${SLEEP_SEC:-1}

该提问来源于开源项目:axsh/wakame-vdc

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

32条回答

  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/rpmbuild

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/rspec

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/to-dropbox

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/to-s3

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/dummy.smoke

    点赞 评论 复制链接分享
  • weixin_39562752 weixin_39562752 4月前

    Looks good. Will merge if the CI is green. +1

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :x: b1c744b9 failure - wakame-ci/vz.smoke

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前
    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/kvm.smoke.allowed-failure

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/kvm.smoke

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :x: b1c744b9 failure - wakame-ci/rspec

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/vdc-manage

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/backup-cleaner

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/rspec

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/vdc-manage

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/backup-cleaner

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/rpmbuild

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/to-dropbox

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/to-s3

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/dummy.smoke

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/kvm.smoke.allowed-failure

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前
    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/kvm.smoke

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/vz.smoke

    点赞 评论 复制链接分享
  • weixin_39562752 weixin_39562752 4月前

    Rspec failed on the known race condition again: https://github.com/axsh/wakame-vdc/issues/475

    
    13:32:41   1) Dcmgr::Scheduler::IPAddress::Incremental#schedule happy paths when previously assigned addresses have been released reassigns the lowest available address
    13:32:41      Failure/Error: expect(subject).to eq "192.168.0.3"
    13:32:41        
    13:32:41        expected: "192.168.0.3"
    13:32:41             got: "192.168.0.6"
    13:32:41        
    13:32:41        (compared using ==)
    13:32:41      Shared Example Group: "reassign released addresses" called from ./spec/dcmgr/scheduler/ip_address/incremental_spec.rb:32
    13:32:41      # ./spec/dcmgr/scheduler/ip_address/incremental_examples/reassign_released_addresses.rb:18:in `block (3 levels) in <top>'
    </top>

    Why did openvz fail? It is failing often lately.

    点赞 评论 复制链接分享
  • weixin_39768247 weixin_39768247 4月前

    Collector restarted. So it looks alarm event issue same as #584.

    /var/log/wakame-vdc/collector.log:

    
    14:22:16 2015-07-14 14:17:39 JobContext thr=JobWorker[0/1] [INFO]: Job complete 16476f1dc8d9c1602e8f12c3e41d358ff88aca14 (Local ID: 16476f1dc8d9c1602e8f12c3e41d358ff88aca14)[ schedule_instance ]: 0.096018235 sec
    14:22:16 I, [2015-07-14T14:17:54.366856 #2029]  INFO -- AlarmCollector: event_caught: ["i-apilve6z"]
    14:22:16 I, [2015-07-14T14:18:01.859603 #2029]  INFO -- HvaCollector: finalize_instance(i-apilve6z) terminated at=2015-07-14 05:18:01 UTC
    14:22:16 I, [2015-07-14T14:18:01.876232 #2029]  INFO -- EventHook: event caught: ["i-apilve6z"]
    14:22:16 I, [2015-07-14T14:18:01.879143 #2029]  INFO -- AlarmCollector: event_caught: ["i-apilve6z"]
    14:22:16 I, [2015-07-14T14:18:01.881325 #2029]  INFO -- AlarmCollector: event_caught: ["i-apilve6z"]
    14:22:16 2015-07-14 14:18:06 Node thr=#<0x007f4542d93d50> [INFO]: Started : AMQP Server=amqp://127.0.0.1/, ID=collector.master, token=c4e22
    </0x007f4542d93d50>

    /var/log/wakame-vdc/hva.log:

    
    14:22:16 2015-07-14 14:18:01 JobContext thr=JobWorker[0/1] [INFO]: Job complete 3c449e21883cae719853898219e342107b4d7076 (Local ID: 3c449e21883cae719853898219e342107b4d7076)[ delete_alarm ]: 0.071143502 sec
    14:22:16 2015-07-14 14:21:08 ThreadPool thr=InstanceMonitor[0/1] [ERROR]: Caught Isono::NodeModules::RpcChannel::RpcError: timeout
    14:22:16    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.20/lib/isono/node_modules/rpc_channel.rb:466:in `wait'
    14:22:16    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.20/lib/isono/node_modules/rpc_channel.rb:153:in `request'
    14:22:16    /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/node_modules/instance_monitor.rb:31:in `check_instance'
    14:22:16    /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/node_modules/instance_monitor.rb:12:in `block (3 levels) in <instancemonitor>'
    14:22:16    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.20/lib/isono/thread_pool.rb:32:in `call'
    14:22:16    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.20/lib/isono/thread_pool.rb:32:in `block (2 levels) in initialize'
    </instancemonitor>
    点赞 评论 复制链接分享
  • weixin_39562752 weixin_39562752 4月前

    I see. Let's discuss later what when we're going to fix these issues.

    This PR looks good. +1

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/rspec

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/vdc-manage

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/backup-cleaner

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/vdc-manage

    点赞 评论 复制链接分享
  • weixin_39593427 weixin_39593427 4月前

    :white_check_mark: b1c744b9 success - wakame-ci/cli/backup-cleaner

    点赞 评论 复制链接分享

相关推荐