- Status Closed
- Percent Complete
- Task Type Bug Report
- Category Packages
- Assigned To No-one
- Operating System
- Severity Critical
- Priority Very High
- Reported Version
- Due in Version Undecided
-
Due Date
Undecided
- Votes
- Private
Attached to Project: Arch Linux 32
Opened by Andreas Baumann - 17.12.2021
Last edited by Andreas Baumann - 25.02.2022
Opened by Andreas Baumann - 17.12.2021
Last edited by Andreas Baumann - 25.02.2022
FS#220 - [python] completly broken
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/mercurial/policy.py", line 69, in _importfrom fakelocals[modname] = mod = getattr(pkg, modname) File "/usr/lib/python3.9/site-packages/mercurial/pycompat.py", line 307, in w return f(object, sysstr(name), *args)
AttributeError: module ‘mercurial.cext’ has no attribute ‘parsers’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/hg", line 61, in <module> dispatch.run() File "/usr/lib/python3.10/importlib/util.py", line 247, in __getattribute__ self.__spec__.loader.exec_module(self) File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/usr/lib/python3.9/site-packages/mercurial/dispatch.py", line 21, in <module> from .i18n import _ File "/usr/lib/python3.10/importlib/util.py", line 247, in __getattribute__ self.__spec__.loader.exec_module(self) File "/usr/lib/python3.9/site-packages/mercurial/i18n.py", line 123, in <module> if _plain(): File "/usr/lib/python3.9/site-packages/mercurial/i18n.py", line 115, in _plain b'HGPLAIN' not in encoding.environ File "/usr/lib/python3.10/importlib/util.py", line 247, in __getattribute__ self.__spec__.loader.exec_module(self) File "/usr/lib/python3.9/site-packages/mercurial/encoding.py", line 41, in <module> charencode = policy.importmod('charencode') File "/usr/lib/python3.9/site-packages/mercurial/policy.py", line 116, in importmod mod = _importfrom(pn, mn) File "/usr/lib/python3.9/site-packages/mercurial/policy.py", line 71, in _importfrom raise ImportError('cannot import name %s' % modname)
ImportError: cannot import name parsers
for instance when building nss.
Closed by Andreas Baumann
25.02.2022 15:13
Reason for closing: Fixed
Additional comments about closing:
25.02.2022 15:13
Reason for closing: Fixed
Additional comments about closing:
most python modules are rebuild for 3.10
and will hit stable soon, we have to
reopen if to many things break.
So, for instance nss needs gyp to build.
gyp is a console_script generated by python-entrypoints, and that one
seems to be currently broken, so you end up with a gyp package without
the shell script '/usr/bin/gyp' for calling the python function.
This means we get an unknown number of packges with broken entrypoints.
Building gyp yeilds:
/usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'
So distutils is part of python itself, I'll try a rebuild of the python package.
My suspicion is that python-entrypoints was broken while reubuilding python
itself (which would mean yet another dependency cycle there).
python-setuptools shows an error
error: cannot copy tree 'build/scripts-3.10': not a directory
Not sure what this is about.
Also the python-boostrap-setuptools package is way old, is this a problem?
And it shows the same error:
error: cannot copy tree 'build/scripts-3.10': not a directory
We should find the root cause for that probably and not just yolo it
/usr/bin/python -Wignore:The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives:DeprecationWarning /tmp/tmppp7giqaj.py
removing /tmp/tmppp7giqaj.py
This is also not a good sign. So however doesn't import setuptools in the future
will have broken python packages?
Alpine does something similar:
Granted.. for an older version.. but..
python-pip-20.3.4-4-pentium4-build.log:/usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'
I think the key to the problem is that entry_points sort of doesn't work.
In general most Python packages build fine, they just "forget" to build certain
scripts. This is really a good build process!
Mmh. python-entrypoints doesn't get installed?
And even if you install it manually inside the chroot you get:
{'name': 'gyp', 'version': '0.1', 'description': 'Generate Your Projects', 'author': 'Chromium Authors', 'author_email': 'chromium-dev@googlegroups.com', 'url': 'http://code.google.com/p/gyp', 'package_dir': {'': 'pylib'}, 'packages': ['gyp', 'gyp.generator'], 'entry_points': {'console_scripts': ['gyp=gyp:script_main']}, 'script_name': 'setup.py', 'script_args': ['build']}
/usr/lib/python3.10/distutils/dist.py:275: UserWarning: Unknown distribution option: 'entry_points'
So python cannot read the map?
So this can be an error in python-attrs, python-ordered-set, but then shouldn't
more packages break?
python-setuptools 1:57.4.0-6, there is a 59.7.0, released on Dec 18. I wonder. Why is upstream using a version from July?
A certain version deprecates Python2, that would be one explanation..
But I cannot upgrade downstream.. and I doubt it has to do with our problems
here, after checking the changelog..
Spottet:
in upstream setuptools (commit 277f9e9316663245e8b8ea9838da515e4e191217).
Let's try this, maybe we need a pip around while bootstrapping setuptools..
Same trick applied there 8c23a532ff5fdd6da8d6e351acc711d02c10a179:
But this breaks stuff very likely when just yoloing the directory..
a rebuild of python-setuptools and python-entrypoints alone doesn't really show
effect.. trying the proved approach "just rebuild enough Python and hope."
Nothing helps, so, if we cannot get this fixed, this is the end of Archlinux32,
as almost everything needs Python nowadays. It would be helpfull if upstream
would document, how they are rebuilding something like python 3.10, but on the
other hand, that's also a lot of effort to do. If the build process would be fully
automatized, there would be no need to document the build, also an option.
I'm just completely lost and I don't know what's going wrong and I currently lack
the knowhow how to fix it..
I tried to use ensurepip in python, then I rebuilt all modules required by setuptools with that python.
The complete list is:
No luck. I still get 'entry_points' errors.
This doesn't look right:
Doing this on 64-bit results in a big list of registered entry points.
So, on 64-bit /usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info/
is a directory containint an entry_points.txt. On pentium4 we have a funny
text file /usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info
only.
".egg-info format: a file or directory placed adjacent to the project’s code and resources, that directly contains the project’s metadata.
"
So, why it is this different? No clue.
Copying the dir-version into a arch32-chroot works, we have working entry_points for instance for gyp. Now let's find out who installs this egg-info and where
from, it seems to be completely borked..
So, upstream has a directory with egg-info in stable and a file too in
testing and staging.
diff /usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info/PKG-INFO staging64/usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info
21d20
< Requires-Python: >=3.6
So, setup.py install writes only PKG-INFO and not the other files (or copies into a file instead
of a directory!).
Creating a directory by hand before calling 'make install' seems to be an obvious
trick..
..which of course doesn't work..
ok, python itself uses a /usr/lib/python3.10/distutils/command/install_egg_info.py
which installs a file-version. If this is a good thing, I do not know.
This one is called when setuptools installs it's own egg_info (and where
the entry_points.txt gets lost).
I know that this file version doesn't work in gyp afterwards.
What is absolutely not clear to me is why the upstream version in stable
works and if I try to rebuild it in testing and staging I get the same
non-working version.. something must have changed in the meantime..
https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html:
".egg-info format eggs are not required to be a directory; they can just be a .egg-info file that directly contains the PKG-INFO metadata."
fine. All egg-infos in the rebuilt packages are of type file. I don't know what decides this being a good
idea, because this effectively eliminates the entry_points (and many more).
https://wiki.archlinux.org/title/Python_package_guidelines describes exactly this issue
Maybe we have to apply this for setuptools and all requred modules, so that we get
the proper egg-info and not the distutils one..
Tried with python-more-itertools, no effect.
Let's try on setuptools itself..
..no effect.
Slowly I think the old setuptools 57.4.0 version is just completely borked in respect to bootstrapping:
bootstrapping.py says that no bootstrapping is required and there is a bootstrap-egg-info folder
containing PKG-INFO and entry-points.txt which are then only partially installed.
Let's try to create a egg-info directory with some of this bootstrapping information after setup.py install..
Ok. Installing a correct egg-info for setuptools works, but doesn't make install_egg_info to install correct
egg-info for instance for more-itertools..
Can I do a python setup.py –single-version-externally-managed ?
install_egg_info.py exists twice:
- setuptools/_distutils/command/install_egg_info.py: basename = "%s-%s-py%d.%d.egg-info" indicates it creates the file-only version, this is a copy paste action from distutils itself
- setuptools/command/install_egg_info.py: uses pkg_resources to build eggs
So it seems, something decides to use the first version instead the second version when installing
the egg-info data of pyton-setuptools itself.
an strace rather shows that the installed distutils is used:
openat(AT_FDCWD, "/usr/lib/python3.10/distutils/command/pycache/install_egg_info.cpython-310.pyc", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/python3.10/distutils/command/install_egg_info.py", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
which is a file-version.
And which is absolutely correct because we don't have neither of the setuptool versions which could
be used.
So in theory installing python-setuptools once without python modules and once pyth python-setuptools
as makedepend should fix the problem..
..or setuptools always falls back to distutils?
Also, if I;m building setuptools with a bootstrapped setuptools, builds still fall back to the distutils
version..
Using 'make egg-info' and copying it manually into pkgdir works. after thos more-itertools builds
it's egg-info correctly. Now let's try to rebuild python-setuptools without the patch..
yep. looking good. now we should be able to reschedule all python modules and let them build.
Currently my build slaves are all down (machine doesn't come up remotely after updating it), so
there might be a delay till everything is built again.
Let's build those manually and remove also remove all the python-setuptool hacks:
python-appdirs
python-more-itertools
python-ordered-set
python-attrs
python-six
python-packaging
python-wheel
python-pyparsing
python-setuptools
Adding to the list:
python-pip
meson
gyp
python-sphinx
Looking better now, still there are issues:
python-nose
python-sphinx: basically a nightmare of dependencies
python-cryptography: now using Rust instead of C implementing it's thing
It's too soon for Python 3.10 to hit stable..
New problems arising on i486:
Seen when building opencv on a i486 VM.
python 3.10.2 is now in testing, it will soon hit stable..