Allow subprojects to be built for both the build and host machine #12258

dcbaker · 2023-09-15T17:52:07Z

Currently, subprojects can only be built for the host machine. This is obviously not a great situation, as you may need to build dependencies for the build machine. This series fixes that (hopefully for both cmake and native subprojects), but allowing a subproject to be configured twice.

This is accomplished by creating copies of the core datastructures (Build, Environment, and CoreData), with the per-machine attributes set so that both the host and the build point to the overall build configuration. All install targets from build machine subprojects have been dropped, to avoid such unsupported installation. In addition, I've ensured that in a host == build setup only one subproject will be configured, so that in the case of

d = dependency('foo', native : false, allow_fallback : true)
d_build = dependency('foo', native : true, allow_fallback : true)

that d_build is just a reference to d

This also requires tracking a number of things on a per-machine basis that weren't before. Namely program overrides. In general we want to override for the host machine, but in some cases it makes sense to override for the build machine (such as for build only subprojects), so a native keyword has been added. This may be useful in cases where different output is needed for the host and build machine anyway.

I've made an effort to split out boilerplate changes (and there are several of them), to make this easier to review.

I won't be surprised if there's still some bugs in this, I've run it through some cross compilation on my system, but it's a large complicated set of changes.

mesonbuild/build.py

Nomura-RH · 2023-09-16T19:02:49Z

EDIT: my bad, I was on the wrong branch : ) the example actually works

I tried setting up a simple cross compilation project that builds a native executable, and builds a dependency for the executable from a subproject. It didn't work:

Top-level meson.build:

project('myexecutable', 'c')

subproject('mylib')

mylib_dep = dependency('mylib', native: true)

myexecutable = executable('myexecutable', 'main.c', native: true, dependencies: mylib_dep)

test('myexecutable-test', myexecutable)

subprojects/mylib/meson.build:

project('mylib', 'c')

mylib = static_library('mylib', 'main.c')

mylib_dep = declare_dependency(link_with: mylib)

meson.override_dependency('mylib', mylib_dep)

meson log:

The Meson build system
Version: 1.2.0.rc3
Source dir: /c/Users/nomura/Desktop/myexecutable
Build dir: /c/Users/nomura/Desktop/myexecutable/build
Build type: cross build
Project name: myexecutable
Project version: undefined
C compiler for the host machine: /c/Users/nomura/Desktop/myexecutable/tools/mwccarm (mwccarm 3.0.136)
C linker for the host machine: /c/Users/nomura/Desktop/myexecutable/tools/mwldarm mwldarm 2.0.87
C compiler for the build machine: gcc (gcc 11.3.0 "gcc (GCC) 11.3.0")
C linker for the build machine: gcc ld.bfd 2.40
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: arm
Host machine cpu: arm
Target machine cpu family: arm
Target machine cpu: arm

Executing subproject mylib 

mylib| Project name: mylib
mylib| Project version: undefined
mylib| C compiler for the host machine: /c/Users/nomura/Desktop/myexecutable/tools/mwccarm (mwccarm 3.0.136)
mylib| C linker for the host machine: /c/Users/nomura/Desktop/myexecutable/tools/mwldarm mwldarm 2.0.87
mylib| C compiler for the build machine: gcc (gcc 11.3.0 "gcc (GCC) 11.3.0")
mylib| C linker for the build machine: gcc ld.bfd 2.40
mylib| Build targets in project: 1
mylib| Subproject mylib finished.

WARNING: Subproject 'mylib' did not override 'mylib' dependency and no variable name specified
Dependency mylib from subproject subprojects/mylib found: NO

meson.build:5:12: ERROR: Dependency 'mylib' is required but not found.

Does this mean that 'mylib' should have two targets for the same lib, one with native: true and one with native: false, and therefore two declare_dependency() and meson.override_dependency() calls?

docs/markdown/snippets/subproject_and_override_program_native.md

dcbaker · 2023-09-16T20:08:26Z

@Nomura-RH did I understand correctly that you did get it working?

Nomura-RH · 2023-09-16T20:13:25Z

@Nomura-RH did I understand correctly that you did get it working?

Yep.

Nomura-RH · 2023-09-17T09:37:32Z

Ok, this time I'm pretty sure I've found an error : ) Nothing build-breaking though.

Sub-subprojects are reported in the logs as being built for the host machine, even when they are actually being built for the build machine. Here is a slightly more involved version of my previous example to show what I mean:

Top-level meson.build:

project('myexecutable', 'c')

mylib_dep = dependency('mylib', native: true)

myexecutable = executable('myexecutable', 'main.c', native: true, dependencies: mylib_dep)

test('myexecutable-test', myexecutable)

subprojects/mylib/meson.build:

project('mylib', 'c')

mysublib_dep = dependency('mysublib')

mylib = static_library('mylib', 'main.c', dependencies: mysublib_dep)

mylib_dep = declare_dependency(link_with: mylib)

meson.override_dependency('mylib', mylib_dep)

subprojects/mysublib/meson.build:

project('mysublib', 'c')

mysublib = static_library('mysublib', 'main.c')

mysublib_dep = declare_dependency(link_with: mysublib)

meson.override_dependency('mysublib', mysublib_dep)

meson log:

The Meson build system
Version: 1.2.99
Source dir: /c/Users/nomura/Desktop/myexecutable
Build dir: /c/Users/nomura/Desktop/myexecutable/build
Build type: cross build
Project name: myexecutable
Project version: undefined
C compiler for the host machine: /c/Users/nomura/Desktop/myexecutable/tools/mwccarm (mwccarm 3.0.136)
C linker for the host machine: /c/Users/nomura/Desktop/myexecutable/tools/mwldarm mwldarm 2.0.87
C compiler for the build machine: gcc (gcc 11.3.0 "gcc (GCC) 11.3.0")
C linker for the build machine: gcc ld.bfd 2.40
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: arm
Host machine cpu: arm
Target machine cpu family: arm
Target machine cpu: arm
Found pkg-config: /usr/bin/pkg-config (1.9.4)
Did not find CMake 'cmake'
Found CMake: NO
Build-time dependency mylib found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency mylib

Executing subproject mylib for machine: build 

mylib| Project name: mylib (for build machine)
mylib| Project version: undefined
mylib| C compiler for the host machine: gcc (gcc 11.3.0 "gcc (GCC) 11.3.0")
mylib| C linker for the host machine: gcc ld.bfd 2.40
mylib| Found pkg-config: /usr/bin/pkg-config (1.9.4)
mylib| Did not find CMake 'cmake'
mylib| Found CMake: NO
mylib| Run-time dependency mysublib found: NO (tried pkgconfig and cmake)
mylib| Looking for a fallback subproject for the dependency mysublib

Executing subproject mysublib for machine: host

mysublib| Project name: mysublib (for build machine)
mysublib| Project version: undefined
mysublib| C compiler for the host machine: gcc (gcc 11.3.0 "gcc (GCC) 11.3.0")
mysublib| C linker for the host machine: gcc ld.bfd 2.40
mysublib| Build targets in project: 1
mysublib| Subproject mysublib finished.

mylib| Dependency mysublib found: YES undefined (overridden)
mylib| Build targets in project: 2
mylib| Subproject mylib finished.

Dependency mylib found: YES undefined (overridden)
Build targets in project: 3

myexecutable undefined

  Subprojects (for host machine)
    mysublib    : YES

  Subprojects (for build machine)
    mylib       : YES

  User defined options
    Cross files : cross.ini
    Native files: native.ini

Found ninja-1.11.1 at /usr/bin/ninja

The log errors are:

Executing subproject mysublib for machine: host

Subprojects (for host machine)
    mysublib    : YES

dcbaker · 2023-09-18T17:38:59Z

@Nomura-RH oh, good catch. Yeah, because the for-build-only subprojects think that host == build they report that they're building for the host machine. Should be fixed in the latest version (with a test!)

Nomura-RH · 2023-09-18T19:10:13Z

@Nomura-RH oh, good catch. Yeah, because the for-build-only subprojects think that host == build they report that they're building for the host machine. Should be fixed in the latest version (with a test!)

The first error is indeed solved, but the second one is still there (the summary at the end still reports the subsubproject in the list of subprojects for the host machine)

dcbaker · 2023-09-18T19:18:38Z

Yup, I didn't pass the change down a level... Fixed now in both cases

Nomura-RH · 2023-09-18T19:29:29Z

Yup, I didn't pass the change down a level... Fixed now in both cases

Almost there :) Now all subprojects get listed for both machines:

Subprojects (for host machine)
    mylib       : YES
    mysublib    : YES (from mylib)

  Subprojects (for build machine)
    mylib       : YES
    mysublib    : YES

EDIT: maybe this is actually my doing, because I added a subproject() call for mylib in the top-level meson.build, and one for mysublib in mylib. But I had to, because otherwise the project didn't configure anymore:

The Meson build system
Version: 1.2.99
Source dir: C:/Users/fseme/Desktop/myexecutable
Build dir: C:/Users/fseme/Desktop/myexecutable/build
Build type: cross build
Project name: myexecutable
Project version: undefined
C compiler for the host machine: C:/Users/fseme/Desktop/myexecutable/metrowerks/mwccarm.exe (mwccarm 3.0.137)
C linker for the host machine: C:/Users/fseme/Desktop/myexecutable/metrowerks/mwldarm.exe mwldarm 2.0.87
C compiler for the build machine: gcc (gcc 13.1.0 "gcc (Rev7, Built by MSYS2 project) 13.1.0")
C linker for the build machine: gcc ld.bfd 2.40
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
Found pkg-config: C:\msys64\mingw64\bin/pkg-config.EXE (0.29.2)
Did not find CMake 'cmake'
Found CMake: NO
Build-time dependency mylib found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency mylib

Executing subproject mylib for machine: build 

mylib| Project name: mylib (for build machine)
mylib| Project version: undefined
mylib| C compiler for the host machine: gcc (gcc 13.1.0 "gcc (Rev7, Built by MSYS2 project) 13.1.0")
mylib| C linker for the host machine: gcc ld.bfd 2.40
mylib| Found pkg-config: C:\msys64\mingw64\bin/pkg-config.EXE (0.29.2)
mylib| Did not find CMake 'cmake'
mylib| Found CMake: NO
mylib| Run-time dependency mysublib found: NO (tried pkgconfig and cmake)
mylib| Looking for a fallback subproject for the dependency mysublib

Executing subproject mylib:mysublib for machine: build

mysublib| Project name: mysublib (for build machine)
mysublib| Project version: undefined
mysublib| C compiler for the host machine: gcc (gcc 13.1.0 "gcc (Rev7, Built by MSYS2 project) 13.1.0")
mysublib| C linker for the host machine: gcc ld.bfd 2.40
mysublib| Build targets in project: 1
mysublib| Subproject mysublib finished.

mylib| Dependency mysublib from subproject mysublib found: NO (subproject failed to configure)

subprojects/mylib/meson.build:3:15: ERROR: Dependency 'mysublib' is required but not found.

dcbaker · 2023-09-18T19:57:48Z

In the "10 native subproject" I see only the "both" subproject in both, and then "buildtool", "hostp", and "test installs" in the build list. but that calling a subproject from a subproject that was called in the main project is probably worth testing... let me see

dcbaker · 2023-09-18T20:39:09Z

I've added more tests for correct machine reporting, and discovered that output rpaths weren't being set correctly in the process, so there's another bug down

Which is required to make test for installed files work correctly

Which de-duplicates a lot of printing material and makes it easier to change things.

forcing the executor to the build machine doesn't actually work correctly once we allow build subprojects.

Which maps subproject name to version

Currently, the assumption is that a subproject will only be built for the host machine. Instead back the storage with PerMachine objects, which fixes this. This goal of this patch is to extend the storage, not enable per-machine subprojects yet. This uncovers some cases where we should be using the SubProject type, but instead are using a string.

The builddir will not mirror the subdir in this case, and we need to track that.

We need this information later to build both a build and a host subproject separately. It's a lot of boilerplate and has been split out to try to simplify review of the difficult pieces

When a subproject is built for both machines, we need to ensure that the outputs don't clash. To do this we'll put the outputs of build only subprojects in a separate root folder (build.subprojects/). We need special methods for this. Currently, this patch changes nothing in the output, but as more boilerplate kind of code it's been split out.

…ild == host config In order to run a build machine only subproject, we create a new Build, Environment, and CoreData object that have look like a host == build configuration, but that are properly merged back into the originals.

The current situation in Meson is that subprojects are always built for the host machine. This means that while cross compiling, a tool that must be run on the build machine must have all of its dependencies met by the system (pkg-config, cmake, config-tool, etc); but dependencies for the host machine can be built. With this patch subprojects may also be built for the build machine. Nothing should ever be installed from a subproject for the build machine, while cross compiling. However, we we are not cross compiling they are installed, since Meson can't really tell the difference between a host and build dependency.

For the most part, what we had before would work correctly for cross building. There is, however a case where this might not work correctly, which is a project for the host machine that builds a build machine target and uses that as an override (think some kind of transpiler), to make this work correctly we need to add a native keyword argument to the `override_find_program` method.

dcbaker requested review from mensinda, xclaesse and jpakkane as code owners September 15, 2023 17:52

dcbaker force-pushed the submit/subproject-per-machine branch 3 times, most recently from d94a857 to 74de6bf Compare September 15, 2023 20:06

dcbaker added dependencies subprojects/wraps cross labels Sep 15, 2023

dcbaker force-pushed the submit/subproject-per-machine branch 3 times, most recently from dc5f2ce to 2b14ab1 Compare September 15, 2023 21:55

dnicolodi reviewed Sep 16, 2023

View reviewed changes

mesonbuild/build.py Show resolved Hide resolved

Nomura-RH mentioned this pull request Sep 16, 2023

"native: true" not passed to dependency fallback #12245

Open

Nomura-RH reviewed Sep 16, 2023

View reviewed changes

docs/markdown/snippets/subproject_and_override_program_native.md Outdated Show resolved Hide resolved

dcbaker force-pushed the submit/subproject-per-machine branch from 2b14ab1 to 95c8865 Compare September 18, 2023 17:35

dcbaker force-pushed the submit/subproject-per-machine branch 2 times, most recently from 39e6c87 to f162b3b Compare September 18, 2023 19:05

dcbaker force-pushed the submit/subproject-per-machine branch from f162b3b to 00e90f0 Compare September 18, 2023 19:18

dcbaker force-pushed the submit/subproject-per-machine branch from 00e90f0 to a285c82 Compare September 18, 2023 20:38

dcbaker force-pushed the submit/subproject-per-machine branch 2 times, most recently from eb7eafb to dc7e517 Compare September 23, 2023 19:49

dcbaker added this to the 1.3.0 milestone Sep 25, 2023

dcbaker linked an issue Sep 25, 2023 that may be closed by this pull request

"native: true" not passed to dependency fallback #12245

Open

dcbaker mentioned this pull request Oct 9, 2023

Remove include_directories absolute path validation #12281

Open

dcbaker force-pushed the submit/subproject-per-machine branch 2 times, most recently from f121b6a to 48dbe2a Compare October 11, 2023 22:42

dcbaker added 16 commits October 12, 2023 10:16

run_single_test: set the host_c_compiler

785ed97

Which is required to make test for installed files work correctly

interpreter: fix annotations for SubProject

cf34999

utils: allow initial arguments for PerThreeMachineDefaultable

f341eec

interpreter/dependencyfallbacks: use a single method for logging found

477f2a8

Which de-duplicates a lot of printing material and makes it easier to change things.

interpreter/dependencyfallbacks: log the machine the dependency is for

8b07d52

dependencies: Report for machine when logging dependencies found

cdb2e2f

cmake: actually use the right machine

043dfdd

forcing the executor to the build machine doesn't actually work correctly once we allow build subprojects.

build: add annotaiton for Build.subprojects

c9b9269

Which maps subproject name to version

IncludeDirs store build-only-subproject

de54ff8

The builddir will not mirror the subdir in this case, and we need to track that.

build: only store one static linker for build == host setups

6e612da

Store whether a build or a target is from a build machine subproject

294d783

We need this information later to build both a build and a host subproject separately. It's a lot of boilerplate and has been split out to try to simplify review of the difficult pieces

dcbaker force-pushed the submit/subproject-per-machine branch from 48dbe2a to 409b0ce Compare October 12, 2023 17:17

xclaesse modified the milestones: 1.3.0, 1.4.0 Oct 17, 2023

dcbaker mentioned this pull request Nov 30, 2023

Ability to specify a build target as both host and target? #12570

Open

dcbaker mentioned this pull request Jan 21, 2024

cmake: Subproject does not use native-compilation for targets #12761

Open

This was referenced Mar 15, 2024

Fix subprojects logic in cross build scenarios #12822

Closed

Allow subprojects to be built for both the build and host machine #12994

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow subprojects to be built for both the build and host machine #12258

Allow subprojects to be built for both the build and host machine #12258

dcbaker commented Sep 15, 2023

Nomura-RH commented Sep 16, 2023 •

edited

Loading

dcbaker commented Sep 16, 2023

Nomura-RH commented Sep 16, 2023

Nomura-RH commented Sep 17, 2023

dcbaker commented Sep 18, 2023

Nomura-RH commented Sep 18, 2023

dcbaker commented Sep 18, 2023

Nomura-RH commented Sep 18, 2023 •

edited

Loading

dcbaker commented Sep 18, 2023

dcbaker commented Sep 18, 2023

Allow subprojects to be built for both the build and host machine #12258

Are you sure you want to change the base?

Allow subprojects to be built for both the build and host machine #12258

Conversation

dcbaker commented Sep 15, 2023

Nomura-RH commented Sep 16, 2023 • edited Loading

dcbaker commented Sep 16, 2023

Nomura-RH commented Sep 16, 2023

Nomura-RH commented Sep 17, 2023

dcbaker commented Sep 18, 2023

Nomura-RH commented Sep 18, 2023

dcbaker commented Sep 18, 2023

Nomura-RH commented Sep 18, 2023 • edited Loading

dcbaker commented Sep 18, 2023

dcbaker commented Sep 18, 2023

Nomura-RH commented Sep 16, 2023 •

edited

Loading

Nomura-RH commented Sep 18, 2023 •

edited

Loading