weixin_39689347
weixin_39689347
2020-12-02 21:36

ansible-lint fails to install from source: AttributeError: 'SpecifierSet' object has no attribute 'split'

Due to the complex workarounds collected in setup.py over the years ansible-lint may fail to install with something like:


pip install .                                                                                                                                                           [13:40:59]
Processing /Users/ssbarnea/.cache/pre-commit/repo5hdz7os2
ERROR: Command errored out with exit status 1:
command: /Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/setup.py'"'"'; __file__='"'"'/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/pip-egg-info
cwd: /private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/
Complete output (21 lines):
/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/git.py:68: UserWarning: "/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt" is shallow and may cause errors
warnings.warn('"%s" is shallow and may cause errors' % (wd.path,))
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/setup.py", line 150, in <module>
__name__ == '__main__' and setuptools.setup(**setup_params)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/Users/ssbarnea/.pyenv/versions/3.7.4/lib/python3.7/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/dist.py", line 447, in __init__
k: v for k, v in attrs.items()
File "/Users/ssbarnea/.pyenv/versions/3.7.4/lib/python3.7/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/dist.py", line 735, in finalize_options
ep.load()(self, ep.name, value)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/dist.py", line 291, in check_specifier
packaging.specifiers.SpecifierSet(value)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/_vendor/packaging/specifiers.py", line 594, in __init__
specifiers = [s.strip() for s in specifiers.split(",") if s.strip()]
AttributeError: 'SpecifierSet' object has no attribute 'split'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
</module></module></string>

It happens that I already encountered this error with two other packages I maintain and the fix was to obliterate all workarounds from setup.py and go for safe setup.cfg approach.

References:

  • https://github.com/pycontribs/selinux/blob/master/setup.py
  • https://github.com/pycontribs/pytest-molecule/blob/master/setup.py
  • https://github.com/pypa/setuptools/issues/1869 - same error another project
  • https://bugs.launchpad.net/tripleo/+bug/1848512 - OpenStack Jobs broken by this bug

This is not fixed until: * a new tag is added as pre-commit requires a tag (any tag) * a release is made on pypi, so others can use it (even a new pre-release should do)

该提问来源于开源项目:ansible/ansible-lint

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

22条回答

  • weixin_39691968 weixin_39691968 4月前

    there is no such thing as a list of supported envs

    I was told that it's the same as Ansible Engine/Core

    点赞 评论 复制链接分享
  • weixin_39689347 weixin_39689347 4月前

    While I do understand that the decision to drop the hacks needs to be measured well, I do not understand why the simple fix from https://github.com/ansible/ansible-lint/pull/591 was not merged and was delayed.

    The action here is to wait for the decision whether we can drop support for older envs.

    How long until we get a decision? It would also be great to have some names of those that can make decisions. The reality is CI (azure) is broken on master and I did not see any PRs done from cores in attempt to fix it.

    Regarding suggestion on #2 : pre-commit by design is installing all hooks using git clone, it cannot do it from wheels (at least not in native way). There is a way to define hooks that install wheels but that would break the ability to auto-update, not to mention that is likely to make execution unpredictable if user forgets to pin wheel version. Mainly I see #2 as a proposal to drop pre-commit support because current codebase no longer installs from source. I personally do not see it acceptable, especially as the issue is not caused by a bug in pre-commit or setuptools, it is solely coming from ansible-lint source code.

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    How long until we get a decision?

    No idea.

    some names

    This is more relevant to since he is on Galaxy team while I'm more focused on the Core Engine related stuff. I'm just volunteering here because there's nobody else with the same level of packaging knowledge, trying to prevent things from breaking badly.

    it cannot do it from wheels

    There's Anthony for that.

    I was thinking about something along the lines of https://github.com/pre-commit/pre-commit-hooks/blob/master/setup.cfg#L26-L31. This would allow us to decouple hook from the tool and potentially make it download less unnecessary stuff + pointing to wheels seems more reliable.

    We could also ask to add native support for pre-commit hooks that won't require using users' machines to build dists.

    点赞 评论 复制链接分享
  • weixin_39955149 weixin_39955149 4月前

    there is a way, it's just a bad idea

    arguably a good thing happened here, we found out that the packaging of ansible-lint was broken pretty early

    with shallow clones the amount of stuff saved isn't really impactful

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    the point is that I want to avoid pip building things where possible. If there was a repo with just a config but without setup.py, it'd be nice: I want that "default" flow that exists already to have pip use additional_dependencies instead of pip install .. So basically, I'm seeing for some dont_build_from_setup_py: true in the config, no other changes needed.

    点赞 评论 复制链接分享
  • weixin_39955149 weixin_39955149 4月前

    please read the thread above, this is addressed

    点赞 评论 复制链接分享
  • weixin_39717318 weixin_39717318 4月前

    I just ran into the same error with

    
    Python 3.7.5
    >>> import setuptools; setuptools.__version__
    '39.0.1'
    >>> import virtualenv; virtualenv.__version__
    '16.7.6'
    >>> import pip; pip.__version__
    '19.3.1'
    

    Quick-Fix which worked for me:

    
    pip install virtualenv==16.3.0 
    
    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    consider testing alpha pre-release v4.1.1a2 since we've dropped support for old setuptools and pip.

    点赞 评论 复制链接分享
  • weixin_39626211 weixin_39626211 4月前

    Hi , is there an ETA of 4.1.1 release? There is an open issue in 4.1.1 milestone, but I'm not sure that it is a real blocker: https://github.com/ansible/ansible-lint/issues?q=is%3Aopen+is%3Aissue+milestone%3A4.1.1

    Thanks!

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    It's not my call to make. Ask .

    点赞 评论 复制链接分享
  • weixin_39729837 weixin_39729837 4月前

    I just ran into the same error with

    
    Python 3.7.5
    <blockquote>
    <blockquote>
    <blockquote>
    <p>import setuptools; setuptools.<strong>version</strong>
    '39.0.1'
    import virtualenv; virtualenv.<strong>version</strong>
    '16.7.6'
    import pip; pip.<strong>version</strong>
    '19.3.1'
    </p></blockquote></blockquote></blockquote>

    Quick-Fix which worked for me:

    
    pip install virtualenv==16.3.0 
    

    just did some tests and 16.7.5 seems the last version to work with this.

    点赞 评论 复制链接分享
  • weixin_39689347 weixin_39689347 4月前

    To be honest, I am very aware of a bunch of broken virtualenv releases and unless you can reproduce the bug with the latest virtualenv version, I doubt anyone would step in to investigate.

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    I'm pretty sure he was installing some old version that had the setup.py shim.

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    as pre-commit requires a tag

    this is not true. you can use any "commitish", including SHA

    点赞 评论 复制链接分享
  • weixin_39689347 weixin_39689347 4月前

    Almost forgot that. Yep could work. The downside is that pre-commit autoupdate will not work.

    Btw, here is one of the failures we got on openstack projects: http://logs.rdoproject.org/58/23158/1/check/tox-linters/e49d829/job-output.txt

    点赞 评论 复制链接分享
  • weixin_39600616 weixin_39600616 4月前

    also encountered this with using ansible-lint as pre-commit hook in a github action job:

    
    2019-10-17T15:45:50.7571288Z ##[group]Run pre-commit run -a
    2019-10-17T15:45:50.7571436Z [36;1mpre-commit run -a[0m
    2019-10-17T15:45:50.7613250Z shell: /bin/bash -e {0}
    2019-10-17T15:45:50.7613418Z env:
    2019-10-17T15:45:50.7613540Z   pythonLocation: /opt/hostedtoolcache/Python/3.7.4/x64
    2019-10-17T15:45:50.7613664Z ##[endgroup]
    2019-10-17T15:45:51.6659471Z [INFO] Initializing environment for https://github.com/IamTheFij/ansible-pre-commit.
    2019-10-17T15:45:52.0082299Z [INFO] Initializing environment for https://github.com/ansible/ansible-lint.
    2019-10-17T15:45:52.6020328Z [INFO] Initializing environment for https://github.com/prettier/prettier.
    2019-10-17T15:45:54.1017508Z [INFO] Initializing environment for https://github.com/adrienverge/yamllint.
    2019-10-17T15:45:54.4985598Z [INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks.
    2019-10-17T15:45:54.9426053Z [INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks:shellcheck.
    2019-10-17T15:45:55.2525741Z [INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
    2019-10-17T15:45:55.6409577Z [INFO] Installing environment for https://github.com/ansible/ansible-lint.
    2019-10-17T15:45:55.6410798Z [INFO] Once installed this environment will be reused.
    2019-10-17T15:45:55.6411247Z [INFO] This may take a few minutes...
    2019-10-17T15:46:01.7001103Z An unexpected error has occurred: CalledProcessError: Command: ('/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/bin/python', '/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/bin/pip', 'install', '.')
    2019-10-17T15:46:01.7002490Z Return code: 1
    2019-10-17T15:46:01.7002927Z Expected return code: 0
    2019-10-17T15:46:01.7003304Z Output: 
    2019-10-17T15:46:01.7004022Z     Processing /home/runner/.cache/pre-commit/repoz8r2td9h
    2019-10-17T15:46:01.7004515Z     
    2019-10-17T15:46:01.7005248Z Errors: 
    2019-10-17T15:46:01.7005676Z         ERROR: Command errored out with exit status 1:
    2019-10-17T15:46:01.7007567Z          command: /home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-x17f_u4z/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-x17f_u4z/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-x17f_u4z/pip-egg-info
    2019-10-17T15:46:01.7008597Z              cwd: /tmp/pip-req-build-x17f_u4z/
    2019-10-17T15:46:01.7009075Z         Complete output (21 lines):
    2019-10-17T15:46:01.7009940Z         /tmp/pip-req-build-x17f_u4z/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/git.py:68: UserWarning: "/tmp/pip-req-build-x17f_u4z" is shallow and may cause errors
    2019-10-17T15:46:01.7010756Z           warnings.warn('"%s" is shallow and may cause errors' % (wd.path,))
    2019-10-17T15:46:01.7011249Z         Traceback (most recent call last):
    2019-10-17T15:46:01.7032531Z           File "<string>", line 1, in <module>
    2019-10-17T15:46:01.7058714Z           File "/tmp/pip-req-build-x17f_u4z/setup.py", line 150, in <module>
    2019-10-17T15:46:01.7059229Z             __name__ == '__main__' and setuptools.setup(**setup_params)
    2019-10-17T15:46:01.7059816Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    2019-10-17T15:46:01.7060042Z             return distutils.core.setup(**attrs)
    2019-10-17T15:46:01.7060255Z           File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/distutils/core.py", line 108, in setup
    2019-10-17T15:46:01.7060453Z             _setup_distribution = dist = klass(attrs)
    2019-10-17T15:46:01.7061175Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/dist.py", line 447, in __init__
    2019-10-17T15:46:01.7061663Z             k: v for k, v in attrs.items()
    2019-10-17T15:46:01.7064777Z           File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/distutils/dist.py", line 292, in __init__
    2019-10-17T15:46:01.7065031Z             self.finalize_options()
    2019-10-17T15:46:01.7065732Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/dist.py", line 735, in finalize_options
    2019-10-17T15:46:01.7065947Z             ep.load()(self, ep.name, value)
    2019-10-17T15:46:01.7066662Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/dist.py", line 291, in check_specifier
    2019-10-17T15:46:01.7067655Z             packaging.specifiers.SpecifierSet(value)
    2019-10-17T15:46:01.7068297Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/_vendor/packaging/specifiers.py", line 594, in __init__
    2019-10-17T15:46:01.7068568Z             specifiers = [s.strip() for s in specifiers.split(",") if s.strip()]
    2019-10-17T15:46:01.7069001Z         AttributeError: 'SpecifierSet' object has no attribute 'split'
    2019-10-17T15:46:01.7069391Z         ----------------------------------------
    2019-10-17T15:46:01.7069583Z     ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    2019-10-17T15:46:01.7069767Z     
    2019-10-17T15:46:01.7069870Z 
    2019-10-17T15:46:01.7070206Z Check the log at /home/runner/.cache/pre-commit/pre-commit.log
    2019-10-17T15:46:01.7223163Z ##[error]Process completed with exit code 1.
    </module></module></string>

    😕

    点赞 评论 复制链接分享
  • weixin_39689347 weixin_39689347 4月前

    Until we get the PRs reviewed, merged and release please use this workaround:

    
    deps =
        # workaround for https://github.com/ansible/ansible-lint/issues/590
        virtualenv==16.3.0 # 16.7.6 not working
        pre-commit
    

    Mainly this is forcing pre-commit to use a version of virtualenv that has an older setuptools, one that is still compatible with ansible-lint.

    点赞 评论 复制链接分享
  • weixin_39673601 weixin_39673601 4月前

    Heya,

    Also hitting it, it's blocking tripleo-validations repository changes.....

    
    2019-10-18 08:19:50.253059 | ubuntu-bionic | Errors:
    2019-10-18 08:19:50.253189 | ubuntu-bionic |         ERROR: Command errored out with exit status 1:
    2019-10-18 08:19:50.253940 | ubuntu-bionic |          command: /home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-q4STID/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-q4STID/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-q4STID/pip-egg-info
    2019-10-18 08:19:50.254064 | ubuntu-bionic |              cwd: /tmp/pip-req-build-q4STID/
    2019-10-18 08:19:50.254160 | ubuntu-bionic |         Complete output (19 lines):
    2019-10-18 08:19:50.254268 | ubuntu-bionic |         Traceback (most recent call last):
    2019-10-18 08:19:50.254368 | ubuntu-bionic |           File "<string>", line 1, in <module>
    2019-10-18 08:19:50.254540 | ubuntu-bionic |           File "/tmp/pip-req-build-q4STID/setup.py", line 150, in <module>
    2019-10-18 08:19:50.254692 | ubuntu-bionic |             __name__ == '__main__' and setuptools.setup(**setup_params)
    2019-10-18 08:19:50.254923 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
    2019-10-18 08:19:50.255022 | ubuntu-bionic |             return distutils.core.setup(**attrs)
    2019-10-18 08:19:50.255143 | ubuntu-bionic |           File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    2019-10-18 08:19:50.255243 | ubuntu-bionic |             _setup_distribution = dist = klass(attrs)
    2019-10-18 08:19:50.255461 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/dist.py", line 447, in __init__
    2019-10-18 08:19:50.255543 | ubuntu-bionic |             k: v for k, v in attrs.items()
    2019-10-18 08:19:50.255668 | ubuntu-bionic |           File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
    2019-10-18 08:19:50.255746 | ubuntu-bionic |             self.finalize_options()
    2019-10-18 08:19:50.256002 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/dist.py", line 735, in finalize_options
    2019-10-18 08:19:50.256095 | ubuntu-bionic |             ep.load()(self, ep.name, value)
    2019-10-18 08:19:50.256323 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/dist.py", line 291, in check_specifier
    2019-10-18 08:19:50.256418 | ubuntu-bionic |             packaging.specifiers.SpecifierSet(value)
    2019-10-18 08:19:50.256669 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/_vendor/packaging/specifiers.py", line 594, in __init__
    2019-10-18 08:19:50.256800 | ubuntu-bionic |             specifiers = [s.strip() for s in specifiers.split(",") if s.strip()]
    2019-10-18 08:19:50.256918 | ubuntu-bionic |         AttributeError: 'SpecifierSet' object has no attribute 'split'
    2019-10-18 08:19:50.257007 | ubuntu-bionic |         ----------------------------------------
    2019-10-18 08:19:50.257201 | ubuntu-bionic |     ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    </module></module></string>
    点赞 评论 复制链接分享
  • weixin_39630182 weixin_39630182 4月前

    According to the setuptools documentation, using setuptools.setup() will automatically load the setup.cfg file. The only gotcha is that it requires setuptools 30.3.0 which will not work Enterprise Linux 7 using the default python-setuptools RPM.

    tl;dr? Using this for the ansible-lint setup.py allows me to install from git.

    
    import setuptools
    
    setuptools.setup()
    

    Any reason why the setup.py is so complicated/hacky? For handling older versions of setuptools we could manually load the configuration using ini and then pass it to setuptools.setup().

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    I'll tell you more: with the latest Pip and pyproject.toml you can even have setup.cfg w/o a setup.py existing in the repo at all.

    But because of the list of supported envs we cannot rely on the latest versions of runtime pre-requisites.

    That "hacks" in setup.py are only there to facilitate the backward compatibility.

    Regarding "it won't work on RHEL": ideally, users should only install from wheels (or in case of RPMs, users should get pre-built artifacts anyway). This issue only hits cases where users build dist locally.

    点赞 评论 复制链接分享
  • weixin_39689347 weixin_39689347 4月前

    But because of the list of supported envs we cannot rely on the latest versions of runtime pre-requisites.

    AFAIK there is no such thing as a list of supported envs, or if exists is clearly not in this repository.

    In the absence of an official list I would assume that CI/CD coverage provides that list.

    Considering that in the last two weeks the CI of the project was broken because of this bug and that any of the two proposed patches is fixing it I would say that at this moment there is no tangible proof that this change breaks more than it fixes.

    Ansible lint is also a pre-commit hook, and pre-commit is always installing from source. Mainly that is what is broken now. Any user using ansible-lint via pre-commit will encounter this bug.

    As this seems to take more and more time to get fix, we may even end-up being forced to fork ansible-lint openstack, example https://review.opendev.org/#/c/689719/

    点赞 评论 复制链接分享
  • weixin_39691968 weixin_39691968 4月前

    1) Back when I was setting up packaging, I got some pushback regarding dropping envs so I'm not doing it w/o an explicit confirmation from Galaxy folks that it's fine. IMHO it's a tool for CIs and dev envs where the runtime is fully under control of the user so they may set up any required prerequisites that we'd ask them to. So yes, I understand why you want this and I'm not opposed to this idea as long as others are okay with this. 2) Just to reiterate, I think I mentioned it somewhere already, I'm sure that it's better to move the pre-commit hook definition into a separate repository and point it to PyPI so that it'd mostly hit wheels. (This would really solve a lot of problems)

    点赞 评论 复制链接分享