Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding support for editable namespace packages #29

Closed
wants to merge 17 commits into from
Closed

Adding support for editable namespace packages #29

wants to merge 17 commits into from

Conversation

Thf772
Copy link
Contributor

@Thf772 Thf772 commented Sep 19, 2023

Namespace packages installed as editable have no __file__ which causes a crash when installing Extensions and DSOs that depend on other DSOs located insde them. Now instead of only looking at the root package to find a DSO, we look for the first package with a non-None __file__.

Fixes #28

@mdavidsaver mdavidsaver mentioned this pull request Sep 20, 2023
Closed
@mdavidsaver
Copy link
Member

@Thf772 Are you able to enable github actions runs in your personal repository? I don't want my having to (re)approve GHA runs to be an impediment to you getting feedback.

@Thf772
Copy link
Contributor Author

Thf772 commented Oct 3, 2023 •
edited
Loading

I've tried setting it up, but apparently I can't run it manually so I don't know if it'll work

Edit: it's working, the workflow is running

@mdavidsaver
Copy link
Member

Understood. I will try to check more often to see when GH insists that I re-approve...

@Thf772
Copy link
Contributor Author

Thf772 commented Oct 3, 2023

Thanks, apparently workflows work in my fork so it should not be as important. Thank you for the help so far. Also apparently you were right about __file__ not existing on namespace packages in older Pythons, the Python 3.6 test failed because of that.

@Thf772
Copy link
Contributor Author

Thf772 commented Oct 3, 2023 •
edited
Loading

Alright, finally I managed to fix the workflow completely. There was another weird bug and I had to dig far too deep and too greedily into Setuptools code to understand...

Basically: when building in editable mode, build_dso is called as a subcommand of build_ext, but build_ext doesn't pass the inplace flag automatically. This means the DSO doesn't get copied into the package directory, and it causes a "File not found" when trying to load it. I don't know why it only causes a crash on MacOS, but at least the problem should be solved.

@mdavidsaver
Copy link
Member

Partially squashed and merged as of 751e45c. I also made some additional changes in 9db427e which are mostly to add comments, but also include some reorganizing.

@mdavidsaver mdavidsaver closed this Oct 8, 2023
@mdavidsaver
Copy link
Member

... I had to dig far too deep and too greedily into Setuptools code to understand...

An masterful summary of my entire experience with setuptools!

@mdavidsaver mdavidsaver added the enhancement New feature or request label Oct 8, 2023
NexediGitlab pushed a commit to SlapOS/slapos that referenced this pull request May 22, 2024
@navytux
stack/slapos: v↑ setuptools-dso 2.9 -> 2.10
5667603 
With newer setuptools that is coming via https://lab.nexedi.com/nexedi/slapos/-/merge_requests/1550 (44.1.1 -> 67.8.0)
we will need a fix from setuptools-dso 2.10 to handle `python setup develop` well:

Previously with setutools-dso 2.9 and setuptools 67.8.0 built shared libraries were not
copied back into the working tree upon develop install which made anything
using pygolang to fail as e.g.

    $ ../bin/gpython
    Traceback (most recent call last):
      File ".../../bin/gpython", line 20, in <module>
        sys.exit(gpython.main())
      File ".../pygolang/gpython/__init__.py", line 456, in main
        pymain(argv, init)
      File ".../pygolang/gpython/__init__.py", line 223, in pymain
        init()
      File ".../pygolang/gpython/__init__.py", line 447, in init
        import golang
      File ".../pygolang/golang/__init__.py", line 41, in <module>
        from golang._gopath import gimport  # make gimport available from golang
      File ".../pygolang/golang/_gopath.py", line 65, in <module>
        from golang import sync
      File ".../pygolang/golang/sync.py", line 36, in <module>
        from golang._sync import \
    ImportError: liblibgolang.so.0.1: cannot open shared object file: No such file or directory

See epics-base/setuptools_dso#29 (comment)
and epics-base/setuptools_dso@2fdf75f2 for details.

P.S. NOTE that changing version of a setup-egg required egg currently does
_not_ force a rebuild. In other words pushing this change into testnodes won't
make pygolang t o be rebuilt at all. I think this is simply a bug in
slapos.buildout to fix.

/reported-by @xavier_thompson
/cc @jerome, @tomo, @Kazuhiko
NexediGitlab pushed a commit to SlapOS/slapos that referenced this pull request May 23, 2024
@navytux
stack/slapos: v↑ setuptools-dso 2.9 -> 2.10
5eed3c4 
With newer setuptools that is coming via https://lab.nexedi.com/nexedi/slapos/-/merge_requests/1550 (44.1.1 -> 67.8.0)
we will need a fix from setuptools-dso 2.10 to handle `python setup develop` well:

Previously with setutools-dso 2.9 and setuptools 67.8.0 built shared libraries were not
copied back into the working tree upon develop install which made anything
using pygolang to fail as e.g.

    $ ../bin/gpython
    Traceback (most recent call last):
      File ".../../bin/gpython", line 20, in <module>
        sys.exit(gpython.main())
      File ".../pygolang/gpython/__init__.py", line 456, in main
        pymain(argv, init)
      File ".../pygolang/gpython/__init__.py", line 223, in pymain
        init()
      File ".../pygolang/gpython/__init__.py", line 447, in init
        import golang
      File ".../pygolang/golang/__init__.py", line 41, in <module>
        from golang._gopath import gimport  # make gimport available from golang
      File ".../pygolang/golang/_gopath.py", line 65, in <module>
        from golang import sync
      File ".../pygolang/golang/sync.py", line 36, in <module>
        from golang._sync import \
    ImportError: liblibgolang.so.0.1: cannot open shared object file: No such file or directory

See epics-base/setuptools_dso#29 (comment)
and epics-base/setuptools_dso@2fdf75f2 for details.

P.S. NOTE that changing version of a setup-egg required egg currently does
_not_ force a rebuild. In other words pushing this change into testnodes won't
make pygolang t o be rebuilt at all. I think this is simply a bug in
slapos.buildout to fix.

/reported-by @xavier_thompson
/cc @jerome, @tomo, @Kazuhiko
/reviewed-on https://lab.nexedi.com/nexedi/slapos/-/merge_requests/1585
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Editable install fails when an extension depending on a DSO is located in a namespace package
2 participants
@Thf772 @mdavidsaver