Bug? Getting a Setuptools related "_read_utf8_with_fallback is not defined"

[autogris]

Hi. I'm trying to debug an error that emerged since upgrading setuptools to 70.0.0. When running Deluge, it gives me the following error:

Traceback (most recent call last):
  File "/usr/bin/deluge-gtk", line 33, in <module>
    sys.exit(load_entry_point('deluge==2.1.1.dev99', 'gui_scripts', 'deluge-gtk')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/__init__.py", line 63, in start
    Gtk().start()
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/__init__.py", line 44, in start
    from .gtkui import GtkUI
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/gtkui.py", line 61, in <module>
    from .pluginmanager import PluginManager
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/pluginmanager.py", line 12, in <module>
    import deluge.pluginmanagerbase
  File "/usr/lib/python3.11/site-packages/deluge/pluginmanagerbase.py", line 15, in <module>
    import pkg_resources
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3282, in <module>
    @_call_aside
     ^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3295, in _initialize_master_working_set
    working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 589, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 926, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 787, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 816, in _resolve_dist
    env = Environment(self.entries)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1014, in __init__
    self.scan(search_path)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1046, in scan
    for dist in find_distributions(item):
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2091, in find_on_path
    yield from factory(fullpath)
               ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2183, in resolve_egg_link
    return next(dist_groups, ())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2179, in <genexpr>
    resolved_paths = (
                     ^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2167, in non_empty_lines
    for line in _read_utf8_with_fallback(path).splitlines():
                ^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_read_utf8_with_fallback' is not defined

I've tried to recompile Deluge, reinstall all python modules, to no success. The supposedly undefined function is in fact present at python3.11/site-packages/setuptools/unicode_utils.py . Is this a bug on setuptools side or Deluge? I'm on Python 3.11.9

[abravalheri]

Could you please also verify if _read_utf8_with_fallback is defined at the bottom of the /usr/lib/python3.11/site-packages/pkg_resources/__init__.py file?

https://github.com/pypa/setuptools/blob/v70.0.0/pkg_resources/__init__.py#L3342-L33607

[autogris]

Yes, is also there. Maybe a problem with Deluge then?

[abravalheri]

Maybe... But the behaviour is very weird...
Maybe there is some form of cyclic behaviour that doesn't let the rest of the file to be evaluated.

Could you try something just for the sake of debugging?

It is a bit unorthodox, but it is a quick way to check...

Can you try to move the lines in

https://github.com/pypa/setuptools/blob/v70.0.0/pkg_resources%2F__init__.py#L3339-L3367

to the begging of the file, e.g., after the imports?

Will that make your program work?

[autogris]

Yes, the program does work after moving those lines!

[autogris]

Thanks for the help, I will use like this for now and see if don't break anything else. Should I create an issue from this discussion?

[abravalheri]

Yes, I think so. We probably should fix this...

The tricky part will be to understand exactly what is going on...

[NiklasMM]

We also experience this bug when iterating over entry points. Anything I could do to help in reproducing?

[abravalheri]

Hi @NiklasMM, yes please!

We are struggling to create a minimal reproducer.

The problem with the original problem description is that there is too much happening with deluge and it does not help to isolate the root cause of the problem.

If someone could create a bare-bones example (ideally with very simple and made up modules/packages, without dependencies) that presents the same problem, that would help a lot.

We are currently discussing this issue in #4399. Please feel free to post there.

[NiklasMM]

I'll have a closer look and report back over in the issue