weixin_39797686
2020-12-29 02:04 阅读 0

Large imagedata resources causing out of memory error

Before starting a PR, please read our contributor tutorial.

I am running into an out of memory error when trying to upload an image to openstack (~7GB in size). This is happening when using the openstack driver in hashicorp/terraform, but the crash is occurring in gophercloud code.

As per issue #1100 this could have something to do with the imagedata/requests.go function Upload requiring an io.Reader instead of io.ReadSeeker and the openstack driver in hashicorp/terraform needed to calculate an MD5 hash on the data

I have tried on a machine with 8GB of RAM and one with 14GB or RAM. There is a rapid spike in memory usage, visible when viewing resource usage through top, leading immediately to the crash below.


openstack_images_image_v2.base_windows_server_2016_image: Still creating... (50s elapsed)
2019-05-16T05:26:50.481Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: fatal error: runtime: out of memory
2019-05-16T05:26:50.482Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:
2019-05-16T05:26:50.482Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime stack:
2019-05-16T05:26:50.483Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.throw(0x118476b, 0x16)
2019-05-16T05:26:50.483Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/panic.go:608 +0x72
2019-05-16T05:26:50.483Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.sysMap(0xc404000000, 0x1a0000000, 0x1d4bf58)
2019-05-16T05:26:50.483Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/mem_linux.go:156 +0xc7
2019-05-16T05:26:50.484Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.(*mheap).sysAlloc(0x1d32240, 0x1a0000000, 0x0, 0xc0003f1e30)
2019-05-16T05:26:50.484Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/malloc.go:619 +0x1c7
2019-05-16T05:26:50.484Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.(*mheap).grow(0x1d32240, 0xcfc9e, 0x0)
2019-05-16T05:26:50.484Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/mheap.go:920 +0x42
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.(*mheap).allocSpanLocked(0x1d32240, 0xcfc9e, 0x1d4bf68, 0x0)
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/mheap.go:848 +0x337
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.(*mheap).alloc_m(0x1d32240, 0xcfc9e, 0xc000160101, 0x1)
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/mheap.go:692 +0x119
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.(*mheap).alloc.func1()
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/mheap.go:759 +0x4c
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.(*mheap).alloc(0x1d32240, 0xcfc9e, 0xc000000101, 0x433c0f)
2019-05-16T05:26:50.485Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/mheap.go:758 +0x8a
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.largeAlloc(0x19f93a800, 0xc000040100, 0xc0004a4c00)
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/malloc.go:1019 +0x97
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.mallocgc.func1()
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/malloc.go:914 +0x46
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.systemstack(0xc000468a80)
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/asm_amd64.s:351 +0x66
2019-05-16T05:26:50.486Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.mstart()
2019-05-16T05:26:50.487Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/proc.go:1229
2019-05-16T05:26:50.487Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:
2019-05-16T05:26:50.487Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: goroutine 295 [running]:
2019-05-16T05:26:50.487Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.systemstack_switch()
2019-05-16T05:26:50.487Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/asm_amd64.s:311 fp=0xc000386618 sp=0xc000386610 pc=0x455d60
2019-05-16T05:26:50.488Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.mallocgc(0x19f93a800, 0x0, 0x0, 0x0)
2019-05-16T05:26:50.488Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/malloc.go:913 +0x896 fp=0xc0003866b8 sp=0xc000386618 pc=0x40be56
2019-05-16T05:26:50.488Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: runtime.slicebytetostring(0x0, 0xc20056a000, 0x19f93a800, 0x1fffffe00, 0x0, 0x0)
2019-05-16T05:26:50.488Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/runtime/string.go:102 +0x9f fp=0xc0003866e8 sp=0xc0003866b8 pc=0x44493f
2019-05-16T05:26:50.488Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: bytes.(*Buffer).String(...)
2019-05-16T05:26:50.489Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/bytes/buffer.go:66
2019-05-16T05:26:52.006Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: github.com/terraform-providers/terraform-provider-openstack/openstack.(*LogRoundTripper).logRequest(0xc0004c4aa0, 0x132da20, 0xc0004ce008, 0x1186324, 0x18, 0x0, 0x0, 0x0, 0x0)
2019-05-16T05:26:52.007Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-openstack/openstack/types.go:106 +0x241 fp=0xc000386778 sp=0xc0003866e8 pc=0xec4c31
2019-05-16T05:26:52.007Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: github.com/terraform-providers/terraform-provider-openstack/openstack.(*LogRoundTripper).RoundTrip(0xc0004c4aa0, 0xc0000ee000, 0x0, 0x0, 0x0)
2019-05-16T05:26:52.007Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-openstack/openstack/types.go:52 +0x69d fp=0xc0003868a0 sp=0xc000386778 pc=0xec494d
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: net/http.send(0xc0000ee000, 0x1325280, 0xc0004c4aa0, 0x0, 0x0, 0x0, 0xc0004ce090, 0xc000538120, 0xc000306a08, 0x1)
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/net/http/client.go:250 +0x14b fp=0xc000386998 sp=0xc0003868a0 pc=0x6eee5b
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: net/http.(*Client).send(0xc0001c4ff8, 0xc0000ee000, 0x0, 0x0, 0x0, 0xc0004ce090, 0x0, 0x1, 0x10ae060)
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/net/http/client.go:174 +0xfa fp=0xc000386a18 sp=0xc000386998 pc=0x6eeb7a
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: net/http.(*Client).do(0xc0001c4ff8, 0xc0000ee000, 0x0, 0x0, 0x0)
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/net/http/client.go:641 +0x2a8 fp=0xc000386b90 sp=0xc000386a18 pc=0x6f0278
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: net/http.(*Client).Do(0xc0001c4ff8, 0xc0000ee000, 0xc, 0xc000296300, 0xb7)
2019-05-16T05:26:52.008Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/goenv/versions/1.11.5/src/net/http/client.go:509 +0x35 fp=0xc000386bc8 sp=0xc000386b90 pc=0x6eff95
2019-05-16T05:26:52.112Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud.(*ProviderClient).Request(0xc0001c4fc0, 0x117102f, 0x3, 0xc000042320, 0x4a, 0xc0003876a0, 0x0, 0x0, 0x0)
2019-05-16T05:26:52.112Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud/provider_client.go:356 +0x404 fp=0xc000387448 sp=0xc000386bc8 pc=0xc06934
2019-05-16T05:26:52.113Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud.(*ServiceClient).Request(0xc000182320, 0x117102f, 0x3, 0xc000042320, 0x4a, 0xc0003076a0, 0x0, 0xc0003076a0, 0xc000307618)
2019-05-16T05:26:52.113Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud/service_client.go:153 +0x1c2 fp=0xc000387570 sp=0xc000387448 pc=0xc0b1d2
2019-05-16T05:26:52.113Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud.(*ServiceClient).Put(0xc000182320, 0xc000042320, 0x4a, 0x115b460, 0xc0004ce008, 0x0, 0x0, 0xc0003076a0, 0x4ce6a4, 0x1800, ...)
2019-05-16T05:26:52.113Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud/service_client.go:94 +0x114 fp=0xc000387628 sp=0xc000387570 pc=0xc0ab64
2019-05-16T05:26:52.113Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4: github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud/openstack/imageservice/v2/imagedata.Upload(0xc000182320, 0xc00008cab0, 0x24, 0x1326a40, 0xc0004ce008, 0x0, 0x0, 0x0, 0x0, 0x0)
2019-05-16T05:26:52.113Z [DEBUG] plugin.terraform-provider-openstack_v1.18.0_x4:    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-openstack/vendor/github.com/gophercloud/gophercloud/openstack/imageservice/v2/imagedata/requests.go:12 +0x1d4 fp=0xc000387710 sp=0xc000387628 pc=0xd5bef4
---------------------------------  Truncated  ---------------------------------

该提问来源于开源项目:gophercloud/gophercloud

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

9条回答 默认 最新

  • weixin_39797686 weixin_39797686 2020-12-29 02:04

    Having a look through the code myself, the problem seems to be with the use of net/http. passing in the io.Reader for the large file object as body in provider_client.go:317: req, err := http.NewRequest(method, url, body) Calling provider_client.go:356: resp, err := client.HTTPClient.Do(req) Causing a massive use of memory and the crash.

    So presumably net/http is keeping the entire file read from the io.Reader in memory, possible more than once, causing a large image file to quickly eat up all the system resources.

    Is there a way to chunk the upload data? (can data be streamed into net/http?)

    点赞 评论 复制链接分享
  • weixin_39797686 weixin_39797686 2020-12-29 02:04

    Seems this might be of use https://gist.github.com/mattetti/5914158/f4d1393d83ebedc682a3c8e7bdc6b49670083b84#gistcomment-1985369

    点赞 评论 复制链接分享
  • weixin_39892800 weixin_39892800 2020-12-29 02:04

    I've spent several hours trying to reproduce this issue.

    The only way I'm able to is by running Terraform in debug mode with OS_DEBUG turned on. Can you confirm this is what you're doing?

    点赞 评论 复制链接分享
  • weixin_39797686 weixin_39797686 2020-12-29 02:04

    I can confirm, I am running with OS_DEBUG. Strangely I never tried without, as I switched it on while trying to diagnose a generic, unhelpful error message with terraform. That led me to the out of memory problem and I just assumed it was related to the generic error I was getting. Now it looks like they are 2 seperate problems. With this problem stopping me from identifying what my other problem is

    点赞 评论 复制链接分享
  • weixin_39892800 weixin_39892800 2020-12-29 02:04

    Now it looks like they are 2 seperate problems. With this problem stopping me from identifying what my other problem is

    We'll eventually get this yak shaved :)

    I have a patch for this out of memory error that I'll get merged within the next day.

    点赞 评论 复制链接分享
  • weixin_39892800 weixin_39892800 2020-12-29 02:04

    -young https://github.com/terraform-providers/terraform-provider-openstack/pull/755 has been merged into Terraform. If you can compile from source, you should not run into an out of memory issue and can try debugging your original issue. Please let me know if you run into any other problems.

    点赞 评论 复制链接分享
  • weixin_39797686 weixin_39797686 2020-12-29 02:04

    Thank you so much for the quick response. I'm currently rebuilding some of our network architecture, so I can't try the fix yet. But i'll let you know how it goes when I do.

    Cheers.

    点赞 评论 复制链接分享
  • weixin_39892800 weixin_39892800 2020-12-29 02:04

    -young Sure, no problem.

    There's been a new plugin release, so you no longer have to compile from source.

    点赞 评论 复制链接分享
  • weixin_39797686 weixin_39797686 2020-12-29 02:04

    Can confirm there are no more issues when uploading a large imagedata resource while OS_DEBUG=1 TF_LOG=DEBUG is enabled. Strangely the other issue I was facing that forced me to turn debug on has disappeared, but I've changed a lot since then, so was probably an issue on my end.

    Thanks for sorting this out so quickly and getting it merged into the master branch.

    点赞 评论 复制链接分享

相关推荐