• Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Packages
  • Assigned To No-one
  • Operating System
  • Severity Critical
  • Priority Very High
  • Reported Version
  • Due in Version Undecided
  • Due Date Undecided
  • Votes
  • Private
Attached to Project: Arch Linux 32
Opened by Andreas Baumann - 17.12.2021
Last edited by Andreas Baumann - 25.02.2022

FS#220 - [python] completly broken

Traceback (most recent call last):

File "/usr/lib/python3.9/site-packages/mercurial/policy.py", line 69, in _importfrom
  fakelocals[modname] = mod = getattr(pkg, modname)
File "/usr/lib/python3.9/site-packages/mercurial/pycompat.py", line 307, in w
  return f(object, sysstr(name), *args)

AttributeError: module ‘mercurial.cext’ has no attribute ‘parsers’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/bin/hg", line 61, in <module>
  dispatch.run()
File "/usr/lib/python3.10/importlib/util.py", line 247, in __getattribute__
  self.__spec__.loader.exec_module(self)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/lib/python3.9/site-packages/mercurial/dispatch.py", line 21, in <module>
  from .i18n import _
File "/usr/lib/python3.10/importlib/util.py", line 247, in __getattribute__
  self.__spec__.loader.exec_module(self)
File "/usr/lib/python3.9/site-packages/mercurial/i18n.py", line 123, in <module>
  if _plain():
File "/usr/lib/python3.9/site-packages/mercurial/i18n.py", line 115, in _plain
  b'HGPLAIN' not in encoding.environ
File "/usr/lib/python3.10/importlib/util.py", line 247, in __getattribute__
  self.__spec__.loader.exec_module(self)
File "/usr/lib/python3.9/site-packages/mercurial/encoding.py", line 41, in <module>
  charencode = policy.importmod('charencode')
File "/usr/lib/python3.9/site-packages/mercurial/policy.py", line 116, in importmod
  mod = _importfrom(pn, mn)
File "/usr/lib/python3.9/site-packages/mercurial/policy.py", line 71, in _importfrom
  raise ImportError('cannot import name %s' % modname)

ImportError: cannot import name parsers

for instance when building nss.

Closed by  Andreas Baumann
25.02.2022 15:13
Reason for closing:  Fixed
Additional comments about closing:  

most python modules are rebuild for 3.10 and will hit stable soon, we have to reopen if to many things break.

Admin
Andreas Baumann commented on 19.12.2021 08:29

So, for instance nss needs gyp to build.
gyp is a console_script generated by python-entrypoints, and that one
seems to be currently broken, so you end up with a gyp package without
the shell script '/usr/bin/gyp' for calling the python function.

This means we get an unknown number of packges with broken entrypoints.

Admin
Andreas Baumann commented on 19.12.2021 08:39

Building gyp yeilds:

/usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'

warnings.warn(msg)

So distutils is part of python itself, I'll try a rebuild of the python package.
My suspicion is that python-entrypoints was broken while reubuilding python
itself (which would mean yet another dependency cycle there).

Admin
Andreas Baumann commented on 19.12.2021 09:54

python-setuptools shows an error

error: cannot copy tree 'build/scripts-3.10': not a directory

Not sure what this is about.

Also the python-boostrap-setuptools package is way old, is this a problem?
And it shows the same error:

error: cannot copy tree 'build/scripts-3.10': not a directory

We should find the root cause for that probably and not just yolo it :-)

Admin
Andreas Baumann commented on 19.12.2021 09:57

/usr/bin/python -Wignore:The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives:DeprecationWarning /tmp/tmppp7giqaj.py
removing /tmp/tmppp7giqaj.py

This is also not a good sign. So however doesn't import setuptools in the future
will have broken python packages?

Admin
Andreas Baumann commented on 19.12.2021 19:00

Alpine does something similar:

      # Otherwise it complains that build/scripts-3.9 cannot be found
      # no other changes noted
      mkdir -p build/scripts-3.9

Granted.. for an older version.. but..

Admin
Andreas Baumann commented on 19.12.2021 19:05

python-pip-20.3.4-4-pentium4-build.log:/usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'

I think the key to the problem is that entry_points sort of doesn't work.

In general most Python packages build fine, they just "forget" to build certain
scripts. This is really a good build process!

Admin
Andreas Baumann commented on 19.12.2021 19:20

Mmh. python-entrypoints doesn't get installed?

And even if you install it manually inside the chroot you get:

{'name': 'gyp', 'version': '0.1', 'description': 'Generate Your Projects', 'author': 'Chromium Authors', 'author_email': 'chromium-dev@googlegroups.com', 'url': 'http://code.google.com/p/gyp', 'package_dir': {'': 'pylib'}, 'packages': ['gyp', 'gyp.generator'], 'entry_points': {'console_scripts': ['gyp=gyp:script_main']}, 'script_name': 'setup.py', 'script_args': ['build']}
/usr/lib/python3.10/distutils/dist.py:275: UserWarning: Unknown distribution option: 'entry_points'

warnings.warn(msg)

So python cannot read the map?

Admin
Andreas Baumann commented on 19.12.2021 19:21

So this can be an error in python-attrs, python-ordered-set, but then shouldn't
more packages break?

Admin
Andreas Baumann commented on 19.12.2021 19:51

python-setuptools 1:57.4.0-6, there is a 59.7.0, released on Dec 18. I wonder. :-) Why is upstream using a version from July?
A certain version deprecates Python2, that would be one explanation..

But I cannot upgrade downstream.. and I doubt it has to do with our problems
here, after checking the changelog..

Admin
Andreas Baumann commented on 20.12.2021 19:32

Spottet:

# Bootstrap for proper metadata
-  python -m ensurepip

in upstream setuptools (commit 277f9e9316663245e8b8ea9838da515e4e191217).

Let's try this, maybe we need a pip around while bootstrapping setuptools..

Admin
Andreas Baumann commented on 20.12.2021 19:37

Same trick applied there 8c23a532ff5fdd6da8d6e351acc711d02c10a179:

-  # Workaround "error: cannot copy tree 'build/scripts-3.10': not a directory"
-  mkdir -p setuptools-$pkgver/build/scripts-3.10
+  # Bootstrap for proper metadata
+  python -m ensurepip

But this breaks stuff very likely when just yoloing the directory..

Admin
Andreas Baumann commented on 20.12.2021 19:54

a rebuild of python-setuptools and python-entrypoints alone doesn't really show
effect.. trying the proved approach "just rebuild enough Python and hope." :-)

Admin
Andreas Baumann commented on 21.12.2021 19:46

Nothing helps, so, if we cannot get this fixed, this is the end of Archlinux32,
as almost everything needs Python nowadays. It would be helpfull if upstream
would document, how they are rebuilding something like python 3.10, but on the
other hand, that's also a lot of effort to do. If the build process would be fully
automatized, there would be no need to document the build, also an option.

I'm just completely lost and I don't know what's going wrong and I currently lack
the knowhow how to fix it..

Admin
Andreas Baumann commented on 30.12.2021 09:39

I tried to use ensurepip in python, then I rebuilt all modules required by setuptools with that python.
The complete list is:

python (with --ensurepip)
python-appdirs
python-more-itertools
python-ordered-set
python-attrs
python-six
python-packaging
python-wheel
python-pyparsing (with 2to3 hack)
python-setuptools
python
python-setuptools

No luck. I still get 'entry_points' errors.

Admin
Andreas Baumann commented on 30.12.2021 14:51

This doesn't look right:

>>> from importlib.metadata import entry_points
>>> from pprint import pprint
>>> pprint(entry_points())
{}

Doing this on 64-bit results in a big list of registered entry points.

Admin
Andreas Baumann commented on 30.12.2021 15:06

So, on 64-bit /usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info/
is a directory containint an entry_points.txt. On pentium4 we have a funny
text file /usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info
only.

".egg-info format: a file or directory placed adjacent to the project’s code and resources, that directly contains the project’s metadata.
"

So, why it is this different? No clue.

Admin
Andreas Baumann commented on 30.12.2021 15:12

Copying the dir-version into a arch32-chroot works, we have working entry_points for instance for gyp. Now let's find out who installs this egg-info and where
from, it seems to be completely borked..

Admin
Andreas Baumann commented on 30.12.2021 15:47

So, upstream has a directory with egg-info in stable and a file too in
testing and staging.

Admin
Andreas Baumann commented on 30.12.2021 15:49

diff /usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info/PKG-INFO staging64/usr/lib/python3.10/site-packages/setuptools-57.4.0-py3.10.egg-info
21d20
< Requires-Python: >=3.6

So, setup.py install writes only PKG-INFO and not the other files (or copies into a file instead
of a directory!).

Admin
Andreas Baumann commented on 30.12.2021 16:05

Creating a directory by hand before calling 'make install' seems to be an obvious
trick..

Admin
Andreas Baumann commented on 30.12.2021 16:38

..which of course doesn't work..

Admin
Andreas Baumann commented on 30.12.2021 20:53

ok, python itself uses a /usr/lib/python3.10/distutils/command/install_egg_info.py
which installs a file-version. If this is a good thing, I do not know.
This one is called when setuptools installs it's own egg_info (and where
the entry_points.txt gets lost).
I know that this file version doesn't work in gyp afterwards.

What is absolutely not clear to me is why the upstream version in stable
works and if I try to rebuild it in testing and staging I get the same
non-working version.. something must have changed in the meantime..

Admin
Andreas Baumann commented on 31.12.2021 09:21

https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html:

".egg-info format eggs are not required to be a directory; they can just be a .egg-info file that directly contains the PKG-INFO metadata."

fine. All egg-infos in the rebuilt packages are of type file. I don't know what decides this being a good
idea, because this effectively eliminates the entry_points (and many more).

Admin
Andreas Baumann commented on 31.12.2021 10:00

https://wiki.archlinux.org/title/Python_package_guidelines describes exactly this issue

If a package needs setuptools to build due to including executables (which is not supported by distutils), but only imports distutils, then building will raise a warning, and the resulting package will be broken (it will not contain the executables):

/usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'
  warnings.warn(msg)
An upstream bug should be reported. To work around the problem, an undocumented setuptools feature can be used:

# fails because of distutils
python setup.py build

# works by using a setuptools shim
python -m setuptools.launch setup.py build

Maybe we have to apply this for setuptools and all requred modules, so that we get
the proper egg-info and not the distutils one..

Admin
Andreas Baumann commented on 31.12.2021 10:03

Tried with python-more-itertools, no effect.
Let's try on setuptools itself..
..no effect.

Admin
Andreas Baumann commented on 31.12.2021 10:19

Slowly I think the old setuptools 57.4.0 version is just completely borked in respect to bootstrapping:
bootstrapping.py says that no bootstrapping is required and there is a bootstrap-egg-info folder
containing PKG-INFO and entry-points.txt which are then only partially installed.
Let's try to create a egg-info directory with some of this bootstrapping information after setup.py install..

Admin
Andreas Baumann commented on 31.12.2021 10:39

Ok. Installing a correct egg-info for setuptools works, but doesn't make install_egg_info to install correct
egg-info for instance for more-itertools..

Admin
Andreas Baumann commented on 31.12.2021 10:50

Can I do a python setup.py –single-version-externally-managed ?

Admin
Andreas Baumann commented on 01.01.2022 11:34

install_egg_info.py exists twice:
- setuptools/_distutils/command/install_egg_info.py: basename = "%s-%s-py%d.%d.egg-info" indicates it creates the file-only version, this is a copy paste action from distutils itself
- setuptools/command/install_egg_info.py: uses pkg_resources to build eggs

So it seems, something decides to use the first version instead the second version when installing
the egg-info data of pyton-setuptools itself.

Admin
Andreas Baumann commented on 01.01.2022 12:40

an strace rather shows that the installed distutils is used:

openat(AT_FDCWD, "/usr/lib/python3.10/distutils/command/pycache/install_egg_info.cpython-310.pyc", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/python3.10/distutils/command/install_egg_info.py", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3

which is a file-version.

And which is absolutely correct because we don't have neither of the setuptool versions which could
be used.

So in theory installing python-setuptools once without python modules and once pyth python-setuptools
as makedepend should fix the problem..

Admin
Andreas Baumann commented on 01.01.2022 12:47

..or setuptools always falls back to distutils?

Admin
Andreas Baumann commented on 01.01.2022 14:35

Also, if I;m building setuptools with a bootstrapped setuptools, builds still fall back to the distutils
version..

Admin
Andreas Baumann commented on 01.01.2022 20:17

Using 'make egg-info' and copying it manually into pkgdir works. after thos more-itertools builds
it's egg-info correctly. Now let's try to rebuild python-setuptools without the patch..

Admin
Andreas Baumann commented on 01.01.2022 20:29

yep. looking good. now we should be able to reschedule all python modules and let them build.
Currently my build slaves are all down (machine doesn't come up remotely after updating it), so
there might be a delay till everything is built again.

Admin
Andreas Baumann commented on 01.01.2022 20:39

Let's build those manually and remove also remove all the python-setuptool hacks:

python-appdirs
python-more-itertools
python-ordered-set
python-attrs
python-six
python-packaging
python-wheel
python-pyparsing
python-setuptools

Admin
Andreas Baumann commented on 02.01.2022 14:25

Adding to the list:

python-pip
meson
gyp
python-sphinx

Admin
Andreas Baumann commented on 13.01.2022 12:28

Looking better now, still there are issues:

python-cryptography: rust, setuptools
python-automat: WARNING: The wheel package is not available. /usr/bin/python: No module named pip
python-jedi: python-sphinx_rtd_theme
- nodejs-lts-fermium
python-backcall: No module named build
- python-build, forced rebuild, ModuleNotFoundError: No module named 'sphinx_argparse_cli.version'
  - python-sphinx-argparse -> forced rebuild
  - python-sphinx-argparse-cli -> forced
-python-nose:
testing/test_nose.py EEEEEEEEEEEEEEEEEE                                  [ 65%]
testing/test_nose.py EEEEEEEEEEEEEEEEEE                                  [ 65%]

python-nose

python-sphinx: basically a nightmare of dependencies

python-cryptography: now using Rust instead of C implementing it's thing

It's too soon for Python 3.10 to hit stable..

Admin
Andreas Baumann commented on 13.02.2022 12:51

New problems arising on i486:

-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.10.2", minimum required is "3.2") 
-- Found PythonLibs: /lib/libpython3.10.so (found suitable exact version "3.10.2") 
*** stack smashing detected ***: terminated

Seen when building opencv on a i486 VM.

Admin
Andreas Baumann commented on 20.02.2022 07:59

python 3.10.2 is now in testing, it will soon hit stable..

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing