weixin_39620578
weixin_39620578
2021-01-03 14:30

Switching back and forth from terminal to IDE requires rebuilds

Whenever I run a bazel build in the terminal and then switch to intellij and resync the project, it does a full rebuild. It says the options have changed so it's discarding the cache.

Looking at the options it sets, --tool_tag stands out as an obvious one that is in the affects_output section of the Bazel CLI docs, so could that be causing it? There are others as well.

How necessary are these and are there any workarounds or improvements that could be made to make switching back and forth a little easier/faster?

If it sounds like I'm talking crazy talk because this isn't happening for you, I can give more details about the project, if you point me in the right direction to start.

该提问来源于开源项目:bazelbuild/intellij

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

36条回答

  • weixin_39644713 weixin_39644713 4月前

    There's the --action_env flag that's supposed to let you set PATH and other variables to make your build truly hermetic.

    But it looks like most actions just ignore --action_env at the moment: https://github.com/bazelbuild/bazel/issues/3320

    点赞 评论 复制链接分享
  • weixin_39946266 weixin_39946266 4月前

    Ah! Thanks for the pointer. I was able to resolve my original issue by adding this to my bazelrc.

    
    build --incompatible_strict_action_env
    
    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    Yes, it's still reproducible from the other machine. 200+ rules rebuild each iteration going back and forth between syncing in the IDE and building in the system terminal. It's still happening, it seems to happen in the default configuration, and it's very unfun.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    On yet another machine, even switching back and forth between IDE sync and bazel build in the IDE terminal triggers extensive rebuilds.

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    bazel build in the IDE terminal

    The IDE terminal still loads your rc script as an interactive shell. Does that machine have additional environment variables set there?

    What's the exact output that bazel gives when it tries to rebuild? Does it tell you which flag/variable changed?

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    As far as I've seen there is no message indicating that cache is being discarded, it simply rebuilds a lot of things as if they were not there. I've looked pretty closely.

    The only extra things in my rc file are like PS1, GIT_PS1_SHOWDIRTYSTATE, and sourcing bazel-complete.bash.

    I'm not entirely sure, but the items that are getting rebuilt seem to be, or be downstream of, genrule rules. There are some in gflags, some in protobuf, some in my buildfile for yasm, and all of libsodium is a single genrule; these seem to get rebuilt every time. Currently on my laptop (probably the best example, as it's broken and has only publicly available versions of bazel and ubuntu) they rebuild even when going back and forth between the IDE sync and the IDE terminal.

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    You don't even get this message?

    
    WARNING: Running Bazel server needs to be killed, because the startup options are different.
    
    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    Not at all, no.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    Here's some example text I get from the Blaze Console during a sync (this is after syncing, bazel build ... in terminal, syncing, bazel build ... in terminal etc.):

    Syncing project: Sync (incremental)...
    Updating VCS...
    Running Bazel info...
    Command: bazel info --tool_tag=ijwb:CLion --curses=no --color=yes --experimental_ui=no --progress_in_terminal_title=no --
    
    Command: git diff --name-status --no-renames 60d0b2fa4c6fa9a68761ec53d79e71ee6af12316
    
    Computing VCS working set...
    Your working set is empty
    Sync targets from project view:
      //...
    
    Building blaze targets...
    Command: bazel build --tool_tag=ijwb:CLion --keep_going --build_event_binary_file=/tmp/intellij-bep-a27ed71a-c469-4792-b427-d8e138014d7f --nobuild_event_binary_file_path_conversion --curses=no --color=yes --experimental_ui=no --progress_in_terminal_title=no --aspects=//:intellij_info_bundled.bzl%intellij_info_aspect --override_repository=intellij_aspect=/home/widders/.CLion2019.1/config/plugins/clwb/aspect --output_groups=intellij-info-cpp,intellij-info-generic,intellij-info-py,intellij-resolve-cpp,intellij-resolve-py -- //...
    
    INFO: Loading complete.  Analyzing...
    INFO: Loading package: //
    INFO: Found 161 targets...
    INFO: Building...
    [3 / 6] Executing genrule //:gflags_declare_h
    [447 / 464] Executing genrule //third_party/libsodium:build
    INFO: From Executing genrule //third_party/yasm:YASM-VERSION:
    1.3.0
    INFO: From Executing genrule //third_party/yasm:YASM-VERSION [for host]:
    1.3.0
    
    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    If there's any additional information I can provide I'm happy to do so. The //:gflags_declare_h rule is probably a good starting point...

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    We should probably move this over to http://github.com/bazelbuild/bazel, so they can help us figure out what caused the rebuild.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    AH! It's looking like there's a strong possibility that garbage left behind by python virtualenv scripts in $PATH are causing genrule builds to be discarded, and likewise every single thing that transitively depends upon them as well.

    It's a bit odd that it doesn't get logged, but it does look like this is probably the underlying cause. I will check on the laptop when I get home; hopefully it holds true there as well.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    Nope, no luck. I've done what I can to ensure that PATH is exactly the same in both cases and it is still rebuilding genrules on the laptop.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    ..........no, I realized the last thing I'd forgotten today.

    I have a script with CC=clang bazel $@ configured as the bazel binary in and an alias for the same in the terminal (in lieu of toolchains, which are a nightmare to set up), and I had forgotten one of these.

    I could probably file a feature request for genrules to log why they are being rebuilt.

    点赞 评论 复制链接分享
  • weixin_39855658 weixin_39855658 4月前

    I had this problem and found it was indeed the environment $PATH variable.

    By inspecting external/.marker under Bazel's working directory I noticed the value for ENV:PATH changed as I went between IntelliJ and terminal. This appears to trigger a full rebuild. Running IntelliJ from the terminal did not work for me (MacOS Mojave + iTerm + zsh + IntelliJ 2019.1).

    Moving my custom path from .zshrc to /etc/paths.d/ made it consistent.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    Yes, $PATH is one of the rebuild-triggering vars for essentially any genrule, which are often upstream of a lot of expensive compilation.

    点赞 评论 复制链接分享
  • weixin_39817176 weixin_39817176 4月前

    How can I see the path? Bazel info does not show any path. Is there a flag to dump all the info that is used to detect the environment changed.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    If you run a build command with -s (or --subcommands, the long form) it will show you all the commands that are run, which will look like, as an example:

    
    SUBCOMMAND: # //third_party/yasm:genstring_license [action 'Executing genrule //third_party/yasm:genstring_license']
    (cd /home/widders/.cache/bazel/_bazel_widders/abcdefabcdefabcdefabcdefabcdefab/execroot/cc && \
      exec env - \
        PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
      /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/third_party/yasm/genstring license_msg bazel-out/k8-fastbuild/bin/third_party/yasm/libyasm/frontends/yasm/license.c third_party/yasm/libyasm/COPYING')
    

    ...which as you can see shows explicitly setting PATH in a subshell before executing the command.

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    Alternative, which I have actually done: just add env | sort > /tmp/whatisthegenruleenv ; to the genrule command and cat /tmp/whatisthegenruleenv afterwards.

    点赞 评论 复制链接分享
  • weixin_39946266 weixin_39946266 4月前

    Has anyone resolved this? I see the same issue with building protobuf when switching between goland and terminal.

    点赞 评论 复制链接分享
  • weixin_39946266 weixin_39946266 4月前

    fwiw: I didnt experience this in goland 2019.1.x

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    So there's the --explain and --verbose_explanations flags that will tell you why the rebuild occured.

    Just run bazel build --explain=/tmp/explanation.txt --verbose_explanations ..., and take a look at /tmp/explanation.txt, you'll see something like:

    
    Executing action 'Executing genrule //path/to:target': Effective client environment has changed. Now using
      PATH=/usr/local/bin:/usr/bin:/bin:...
    .
    

    It doesn't tell you what your previous PATH was, so that's kind of annoying, but you can run with the two flags in both environments and then compare the differences.

    点赞 评论 复制链接分享
  • weixin_39946266 weixin_39946266 4月前

    Why is the intellij bazel plugin using a different path? I don't understand the resolution.

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    The IDE inherits its PATH from the desktop environment. Your command line is likely the one using a different PATH. We can't control how you modify your PATH in your rc scripts. If you look at what's actually getting added/removed from your PATH, then it could help you narrow down where the change is happening.

    点赞 评论 复制链接分享
  • weixin_39946266 weixin_39946266 4月前

    Isn't there some way to get deterministic builds with bazel? Why does some different in PATH change how protoc is compiled?

    I understand this might be leading out of scope of this repository, but if you could point me in the right direction I would greatly appreciate it.

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    Isn't there some way to get deterministic builds with bazel?

    The build is deterministic, given that the PATH and its contents stays the same. Do you mean hermetic build? I agree depending on the PATH is really not hermetic.

    bazel calls out to some executables on your path for binaries that it doesn't bundle (I think things like cp, mv for genrules). It's just a possibility that it might affect build output, most of the time it doesn't. bazel is just overly eager in assuming a rebuild is necessary. You can even add an empty directory to your PATH and it'll trigger a rebuild (e.g., PATH=$PATH:/bogus/bin bazel build). It really should be something the bazel team should fix.

    点赞 评论 复制链接分享
  • weixin_39946266 weixin_39946266 4月前

    Yes, hermetic builds. Thanks!

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    --tool_tag should not be the culprit.

    Do you have any custom flags in the .bazelproject file? Do you have a different JAVA_HOME on the command line vs in the IDE?

    点赞 评论 复制链接分享
  • weixin_39620578 weixin_39620578 4月前

    No, and I'm using //tools/jdk:absolute_javabase as the --[host_]javabase in a bazelrc to try to avoid the issue.

    点赞 评论 复制链接分享
  • weixin_39792751 weixin_39792751 4月前

    I have noticed in In Goland (pre Bazel Bazel 0.22) it's historically been due to IJ PATH creep from within the IJ terminal.

    点赞 评论 复制链接分享
  • weixin_39644713 weixin_39644713 4月前

    How are you launching the IDE? If you launch it via command line instead of the launcher icon, then it should inherit the same environment variables as bazel on the command line.

    If no other flags are added, then it's probably caused by different environment variables.

    点赞 评论 复制链接分享
  • weixin_39607240 weixin_39607240 4月前

    I also had this issue when launching IntelliJ from outside the terminal. Starting IJ from the terminal fixed it.

    We checked the environment variables, but there aren't any significant difference; so we are still clueless about the root causes.

    点赞 评论 复制链接分享
  • weixin_39607240 weixin_39607240 4月前

    It was indeed due to the difference in the PATH variable that caused the issue for me. I have a few terminal plugins that modify that variable, but only for interactive shell. Resetting PATH to the same value as IntelliJ's also fixed the issue.

    点赞 评论 复制链接分享
  • weixin_39792751 weixin_39792751 4月前

    It appears that there are specific things still added by the Goland IJ terminal that are not present in the external shell's $PATH when launched via the command line. We have modified our workflow so this is not a problem and I believe some newer-bazel defaults regarding action-env may have resolved this.

    点赞 评论 复制链接分享
  • weixin_39620578 weixin_39620578 4月前

    I found a situation where JAVA_HOME actually was different, sorry. Making sure this was unset in both places made it keep the cache. However as I mentioned, I am using :absolute_javabase as my --javabase and --host_javabase specifically to avoid issues like this, so it doesn't seem like it should matter what that environment variable is set to.

    So do you think it's actually a Bazel bug that's causing the invalidation if the environment changes like this since it shouldn't actually matter?

    点赞 评论 复制链接分享
  • weixin_39816260 weixin_39816260 4月前

    This also happens to me in C++ projects. Unless I use the terminal emulator in the tool window inside clion, bazel always discards cached results when switching between terminal and the IDE.

    There are diffs in the env:

    Unset in system terminal: APPDIR APPIMAGE GIO_LAUNCHED_DESKTOP_FILE GIO_LAUNCHED_DESKTOP_FILE_PID OWD TERMINAL_EMULATOR TOOLBOX_VERSION

    Unset in IDE terminal: COLORTERM GNOME_TERMINAL_SCREEN GNOME_TERMINAL_SERVICE VTE_VERSION

    Set in both but empty in IDE terminal: GTK_MODULES

    Set to different value: OLDPWD (pretty sure this is fine :stuck_out_tongue:)

    When I copied the six env values other than OWD in the "Unset in system terminal" category above into a system shell, it stops having this rebuild problem.

    ... and now that I've done that once, even in a new terminal and across calls to clean --expunge I cannot seem to reproduce the problem at the moment.

    It's been an ongoing and especially insidious problem because some of my build rules call into existing non-bazel build processes and take a lot of CPU time to finish, yet will typically rerun every single time I go back and forth between system terminal and IDE sync.

    I'll see if I can figure out how to make it start reproducing again.

    edit: No dice after a reboot, things are continuing to work flawlessly as they have never before. I will check another dev machine.

    点赞 评论 复制链接分享

相关推荐