Fix subprojects logic in cross build scenarios #12822

oleavr · 2024-02-07T23:10:41Z

Where we cannot use the same subproject state for both machines. For example, the subproject built for native: true in such a case should not be cross-compiling. Files written to the build directory also should not clash.

We may not even need to use a subproject for both machines, for example if the build machine has a dependency installed, and we only need to fall back to a subproject for the host machine. This was also not working previously.

Fixes: #10947

tristan957 · 2024-02-08T19:42:32Z

cc @dcbaker

oleavr · 2024-03-11T23:47:02Z

Tried this on a more comprehensive subproject (GLib) today, and found some issues. Will revise this PR as soon as I've got the rough edges sorted. Moving back to Draft state for now.

oleavr · 2024-03-14T12:21:51Z

This should now be ready for review. The CI failures look like the same as on latest master, so I've disregarded them for now.

dcbaker

I'm sorry I didn't see this when Tristian pinged me in February, I get way too many GitHub notifications :(

First, does this allow a subproject to be built for both the host and build machine simultaneously? If not that's a full stop issue for me, I need both, I need that right now, and having an error subproject A configures for the build machine and then tries to also configure for the host machine is worse than the status quo, since it's race to see which one configures first. This comes up a lot with Rust because of proc-macros.

As an overall note, please stop using "native" and "cross", that terminology is confusing and not commonly used, and we're trying to move to "build machine" and "host machine" instead.

I'm going to need more time to look at this and think all of the potential corner cases involved

mesonbuild/backend/backends.py

mesonbuild/backend/ninjabackend.py

mesonbuild/coredata.py

dcbaker · 2024-03-14T16:44:36Z

mesonbuild/interpreter/interpreter.py

@@ -292,7 +309,7 @@ def __init__(
        self.processed_buildfiles: T.Set[str] = set()
        self.project_args_frozen = False
        self.global_args_frozen = False  # implies self.project_args_frozen
-        self.subprojects: T.Dict[str, SubprojectHolder] = {}
+        self.subprojects: T.Dict[T.Tuple[str, MachineChoice], SubprojectHolder] = {}


We have PerMachine for storing things that are per-machine, we should use that here instead of storing it in the dict.

Thanks! Will make this change on my next iteration through, ran out of time this evening.

oleavr · 2024-03-14T18:51:05Z

I'm sorry I didn't see this when Tristian pinged me in February, I get way too many GitHub notifications :(

No worries! I know the feeling all too well 😅

First, does this allow a subproject to be built for both the host and build machine simultaneously?

Yes, that's my exact use-case.

If not that's a full stop issue for me, I need both, I need that right now,

Same!

and having an error subproject A configures for the build machine and then tries to also configure for the host machine is worse than the status quo, since it's race to see which one configures first. This comes up a lot with Rust because of proc-macros.

I believe I retained that logic.

As an overall note, please stop using "native" and "cross", that terminology is confusing and not commonly used, and we're trying to move to "build machine" and "host machine" instead.

Ah yes, not happy about that. I was struggling to come up with a term to describe is_subproject and host_machine is not build_machine and subproject.for_machine is build_machine. Inside such an Interpreter it shouldn't look like a cross build, but Target instances created are always for the build machine since that is shared state. But I found it challenging to pick "shared state with fixups on access" vs. "transformed state with potential merge at the end" for the different parts of the state, and ended up going back and forth a few times. One tricky aspect is that a subproject might have some native: true target that it crafts for the particular host machine, so you can't conflate it with the same target inside the subproject built for the build machine.

I'm going to need more time to look at this and think all of the potential corner cases involved

Much appreciated! 🙏

To be used for the subproject per machine logic.

We need this information later to build both a build and a host subproject separately. It's a lot of boilerplate and has been split out to try to simplify review of the difficult pieces.

When a subproject is built for both machines, we need to ensure that the outputs don't clash. To do this we'll put the outputs of build only subprojects in a separate root folder (build.subprojects/). We need special methods for this. Currently, this patch changes nothing in the output, but as more boilerplate kind of code it's been split out.

mesonbuild/build.py

Where we cannot use the same subproject state for both machines. We may not even need to use a subproject for both machines, for example if the build machine has a dependency installed, and we only need to fall back to a subproject for the host machine. Fixes: mesonbuild#10947

tristan957 · 2024-03-15T02:00:16Z

Looks like there were some timeouts: https://github.com/mesonbuild/meson/actions/runs/8288958897/job/22684490962?pr=12822#step:4:1104. If you have access to linux, you might try running this test in your local environment, just in case.

oleavr · 2024-03-15T21:16:19Z

Oops, apologies, I wasn't aware of #12258 -- it already has better solutions for several things, and some fixes not covered by this PR. I've started reworking the fixes unique to this PR on top of @dcbaker's PR (after rebasing it on master), and will open a new PR soon. Cheers!

oleavr requested a review from jpakkane as a code owner February 7, 2024 23:10

oleavr force-pushed the fix/subproject-cross branch from e78f5c5 to 464f9b3 Compare February 8, 2024 08:49

oleavr marked this pull request as draft February 8, 2024 08:50

oleavr force-pushed the fix/subproject-cross branch 4 times, most recently from 028230b to c32fb7b Compare February 8, 2024 15:13

oleavr force-pushed the fix/subproject-cross branch 2 times, most recently from c1d8a8f to 9762441 Compare February 8, 2024 20:50

oleavr marked this pull request as ready for review February 8, 2024 22:08

oleavr marked this pull request as draft March 11, 2024 23:48

oleavr force-pushed the fix/subproject-cross branch 2 times, most recently from fdac3a7 to 3fd9e08 Compare March 13, 2024 22:57

oleavr marked this pull request as ready for review March 14, 2024 12:15

oleavr requested a review from xclaesse as a code owner March 14, 2024 12:15

dcbaker reviewed Mar 14, 2024

View reviewed changes

oleavr and others added 3 commits March 15, 2024 00:21

coredata: Add is_build_only placeholder

76dc570

To be used for the subproject per machine logic.

Store whether a target is from a build machine subproject

ee9f922

We need this information later to build both a build and a host subproject separately. It's a lot of boilerplate and has been split out to try to simplify review of the difficult pieces.

oleavr force-pushed the fix/subproject-cross branch from 3fd9e08 to 755b128 Compare March 14, 2024 23:21

oleavr requested a review from mensinda as a code owner March 14, 2024 23:21

github-advanced-security bot found potential problems Mar 14, 2024

View reviewed changes

mesonbuild/build.py Fixed Show fixed Hide fixed

oleavr force-pushed the fix/subproject-cross branch from 755b128 to eea60e5 Compare March 14, 2024 23:33

oleavr closed this Mar 15, 2024

oleavr deleted the fix/subproject-cross branch March 19, 2024 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix subprojects logic in cross build scenarios #12822

Fix subprojects logic in cross build scenarios #12822

oleavr commented Feb 7, 2024 •

edited

Loading

tristan957 commented Feb 8, 2024

oleavr commented Mar 11, 2024

oleavr commented Mar 14, 2024

dcbaker left a comment

dcbaker Mar 14, 2024

oleavr Mar 14, 2024 •

edited

Loading

oleavr commented Mar 14, 2024 •

edited

Loading

tristan957 commented Mar 15, 2024

oleavr commented Mar 15, 2024

Fix subprojects logic in cross build scenarios #12822

Fix subprojects logic in cross build scenarios #12822

Conversation

oleavr commented Feb 7, 2024 • edited Loading

tristan957 commented Feb 8, 2024

oleavr commented Mar 11, 2024

oleavr commented Mar 14, 2024

dcbaker left a comment

Choose a reason for hiding this comment

dcbaker Mar 14, 2024

Choose a reason for hiding this comment

oleavr Mar 14, 2024 • edited Loading

Choose a reason for hiding this comment

oleavr commented Mar 14, 2024 • edited Loading

tristan957 commented Mar 15, 2024

oleavr commented Mar 15, 2024

oleavr commented Feb 7, 2024 •

edited

Loading

oleavr Mar 14, 2024 •

edited

Loading

oleavr commented Mar 14, 2024 •

edited

Loading