circuitpython_setboard command python version support

[FoamyGuy]

the circuitpython_setboard command currently supports python 3.10+ only due to a change between py3.9 and py3.10.

Here is an example of the traceback that results from trying a version below 3.10:

$ circuitpython_setboard adafruit_feather_esp32s3_4mbflash_2mbpsram
setting board: adafruit_feather_esp32s3_4mbflash_2mbpsram
Traceback (most recent call last):
  File ".../.venv/bin/circuitpython_setboard", line 8, in <module>
    sys.exit(set_board())
  File ".../.venv/lib/python3.9/site-packages/circuitpython_setboard/__init__.py", line 18, in set_board
    board_stubs_file  =  resources.files("board-stubs").joinpath("__init__.pyi")
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/resources.py", line 147, in files
    return _common.from_package(_get_package(package))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/_common.py", line 14, in from_package
    return fallback_resources(package.__spec__)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/_common.py", line 18, in fallback_resources
    package_directory  =  pathlib.Path(spec.origin).parent
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1072, in __new__
    self  =  cls._from_parts(args, init=False)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 697, in _from_parts
    drv, root, parts  =  self._parse_args(args)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 681, in _parse_args
    a  =  os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

We could improve the situation with one these:

• try to make the code compatible with py3.9. Acording to https://devguide.python.org/versions/#supported-versions py3.9 isn't EoL until October 2025, so it could make sense to try to support it for a while.
• Catch the exception and output a more helpful error message letting the user know about the 3.10+ requirement.

[jepler]

Do you understand which part of the code is dependent on 3.10? importlib.resources.files is listed as being in 3.9

however I found at least one other project having trouble with importlib.resources.files in 3.9, without much explanation: jupyter/notebook#7016

[FoamyGuy]

I don't necessarily understand the reason behind the issue, everything underneath our code in the stack trace is a bit cryptic to me. Something is resolving to None, but it's unclear to me what specific thing is actually None. I searched around for a bit with different parts of the stack trace and was able to find references to the same or similar issues, but nothing really stood out as a cause or way to change the code to make it work.

The crux of it is that resources.files("board-stubs") from here:

circuitpython/tools/board_stubs/circuitpython_setboard/__init__.py

Line 102 in cbd5131

board_stubs_file = resources.files("board-stubs").joinpath("__init__.pyi")

has different behavior on 3.9 vs. 3.10+.

On 3.9 this statement raises the exception with stacktrace above. On 3.10 it returns an object containing the path to the board-stubs/ directory which is installed with circuitpython-stubs.

On py3.10 it returns:

MultiplexedPath('/path/to/py_venv/lib/python3.10/site-packages/board-stubs')

That path is used afterward in order to copy the board file specified by the user into board-stubs/ so that it'll be the one that IDEs and other tools see.

If we can find a different way to programmatically get the path of site-packages it could be used instead of resources.files() in order to work around this problem.

[jepler]

OK, here's the reason. Because it does not contain a init.py, board-stubs is a "namespace package". Support for namespace packages was added in 3.10: python/cpython#86295

pip-installable "importlib-resources" supports namespace packages https://github.com/python/importlib_resources so it could be used as a substitute across any supported python version. I don't know if simply adding board-stubs/__init__.py could be a better solution.

With /tmp/t1/__init__.py and empty directory t2 on python 3.9:

Python 3.9.16 (main, Jun 23 2024, 09:40:19) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.resources import files
>>> files("t1")
PosixPath('/tmp/t1')
>>> files("t2")
... 
TypeError: expected str, bytes or os.PathLike object, not NoneType

[FoamyGuy]

Ahhh I see, thank you for tracking that down and sharing the cause.

I refactored it to use os and site instead of resources.files and didn't see this until I created that PR.

Knowing the real cause now though, I do think adding __init__.py in the necessary directories is probably the best solution.

I will test that locally in a little bit and close that PR and make a new one.

[justmobilize]

I think it depends on what versions of python you want to support. resources.files wasn't added until 3.9

[FoamyGuy]

@justmobilize I don't think we intent to support anything lower than 3.9. Although it seems the issue isn't whether resources.files exists at all, but rather whether it supports those "namespace packages" which changed between 3.9 and 3.10 it seems.

[FoamyGuy]

Knowing the real cause now though, I do think adding init.py in the necessary directories is probably the best solution.
I will test that locally in a little bit and close that PR and make a new one.

I tested that by manually adding a __init__.py inside of both board-stubs/ and board_definitions/. I was worried that the one inside board-stubs/ would take precedent over the __init__.pyi file which is the only other thing in that dir, but inside the IDE it does still correctly show information from the pyi file even when the empty .py file exists.

However it may be a little tricky to create that file inside of board-stubs "for real", I just manually created with touch but we'll need it to happen automatically some time after the directory and pyi file are generated from the text in the comments in shared-bindings. I'm not sure the best place to make that happen from, maybe inside the make stubs command?

[justmobilize]

@FoamyGuy it might be worth adding a version check at the very top of the file (before specific imports), giving a clean error if they are on anything less than 3.9

Or specifying the minimum version so it can't be installed on 3.8 or below

I'm always seeing users with random versions

[FoamyGuy]

Makes sense to me, I added an explicit check version in the new PR.

This one also uses touch to create the __init__.py files inside board-stubs and board_definitions.

[justmobilize]

Awesome!