update synocli-devel

[hgy59]

Description

• update strace to v6.5
• add and fix gdb for qoriq arch
• add llvm-9.0 for qoric arch
• add binutils-2.32 for OLD_PPC_ARCHS

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Package update

[th0ma7]

A few minor adjustments but otherwise looking good!

[hgy59]

@th0ma7 the only thing that bothers me, is the build of native/llvm for all archs.
native/llvm must only be built for packages depending on cross/llvm-9` but it is built for all packages, even ARMv5 that does not include llvm at all.

This is an issue with github actions, since every package is built in its own environment (locally it builds native llvm only once and takes up to 3 h on my older build env).
Even on the github action environment the build of native/llvm takes more than one hour.

For ARMv5 archs the cause is the pre-build-native target that is called before the UNSUPPORTED_ARCHS check (guessing).
For archs using cross/llvm-latest the cause is the OPTIONAL_DEPENDS of cross/llvm-9 in cross/llvm. Optionally it always depends on native/llvm and that is built with the pre-build-native target.

Probably you can remeber why you introduced pre-build-native target in #4200?
If we let make resolve the dependencies, this would not happen, but if we use the dependency-list target that resolves all dependencies (including optional deps.) we do not find the real - arch or gcc version specific - dependencies.

I cannot imagine a solution (exept removing the pre-build-native target), so this will probably not be part of this PR.

[hgy59]

set to WIP:
The spksrc.main-depends.mk is missing the PLIST handling and the generated packages are missing the files of the related packages.

[th0ma7]

@th0ma7 the only thing that bothers me, is the build of native/llvm for all archs. native/llvm must only be built for packages depending on cross/llvm-9` but it is built for all packages, even ARMv5 that does not include llvm at all.

Agreed, the nicest would be to either have a workaround to:

1. only build the needed bits (which I haven't found how) or
2. download a version from somewhere like from a debian package
3. or perhaps installing it by default in the debian container (wouldn't that be the easiest?)

This is an issue with github actions, since every package is built in its own environment (locally it builds native llvm only once and takes up to 3 h on my older build env). Even on the github action environment the build of native/llvm takes more than one hour.

For ARMv5 archs the cause is the pre-build-native target that is called before the UNSUPPORTED_ARCHS check (guessing).

It is.

For archs using cross/llvm-latest the cause is the OPTIONAL_DEPENDS of cross/llvm-9 in cross/llvm. Optionally it always depends on native/llvm and that is built with the pre-build-native target.

A work-around that would be to add a new "phase" within the build process to do the pre-build-native for all and use that resulting build image for every archs. This would avoid building native portions for every archs, but from a build duration perspective that wouldn't change much.

Probably you can remeber why you introduced pre-build-native target in #4200? If we let make resolve the dependencies, this would not happen, but if we use the dependency-list target that resolves all dependencies (including optional deps.) we do not find the real - arch or gcc version specific - dependencies.

That is mandatory for parallel building as otherwise every archs kicks-in at the same time and tries to build the same native dependencies one over the other causing it to fail miserably. Work-around could be:

1. using flock (unsure how I would do that though but probably feasible) similarly to the download code. But when running on your local environment, all arch builds except one will be pending for a really long duration for things like llvm (or legacy cmake for that matter that takes a good 10+ minutes if I recall). Adjusting timer would be a hassle. On the other hand that would solve things for github-action as being independant for every archs.
2. ensuring native is used by all builds on github-action and being cached - eventually even llvm would be cached in there potentially, thus expediting the build. But unsure how to do that neither.

I cannot imagine a solution (exept removing the pre-build-native target), so this will probably not be part of this PR.

Such a change would be relatively intrusive and benefit from being standalone.... Unless we install it into the docker image or find binaries to download from.

EDIT: further looking at things, the real missing part is clang-tblgen which I can't find from the debian archive.

EDIT2: But it may well be available in a linux compatible binary form from the llvm download page https://releases.llvm.org/download.html

[hgy59]

That is mandatory for parallel building as otherwise every archs kicks-in at the same time and tries to build the same native dependencies one over the other causing it to fail miserably. Work-around could be:

IMHO this is not the case. Every build in the matrix runs in its own VM and does not share the native build with other arch-TCVERSION build. The only concurrency problem was, when all VMs wanted to download the same sources, but this we fixed with the prepare job and the caching of downloaded files.
please correct me, if I am wrong...

[hgy59]

EDIT2: But it may well be available in a linux compatible binary form from the llvm download page https://releases.llvm.org/download.html

the only available prebuilt version for x64 (amd64) is for freebsd11
https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/clang+llvm-9.0.1-amd64-unknown-freebsd11.tar.xz

But this archive is missing clang-tblgen too.

What about self hosting the native/llvm-9, once built locally?

[hgy59]

EDIT2: But it may well be available in a linux compatible binary form from the llvm download page https://releases.llvm.org/download.html

the only available prebuilt version for x64 (amd64) is for freebsd11 https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/clang+llvm-9.0.1-amd64-unknown-freebsd11.tar.xz

But this archive is missing clang-tblgen too.

What about self hosting the native/llvm-9, once built locally?

@th0ma7 I started a test to try selfhosted prebuilt native llvm-9.0 (local builds have a slightly different build path (/spksrc/native/llvm-9.0/ vs /github/workspace/native/llvm-9.0/) but library dependencies are relative, so this should not be a problem. 🤞

The prebuilt package is only 55MB tar.gz, so this will be a fast install of native/llvm-9.0.

[th0ma7]

That is mandatory for parallel building as otherwise every archs kicks-in at the same time and tries to build the same native dependencies one over the other causing it to fail miserably. Work-around could be:

IMHO this is not the case. Every build in the matrix runs in its own VM and does not share the native build with other arch-TCVERSION build. The only concurrency problem was, when all VMs wanted to download the same sources, but this we fixed with the prepare job and the caching of downloaded files. please correct me, if I am wrong...

You are right in a github-action environment... But when building locally using make all-supported it's totally different and the concurrency problem hits you real hard. For that the download problem was fixed using flock. On github-action as it spawn multiple VMs we pre-download and mount for each VM afterwards to avoid duplicating data (i.e. different use-case).

So again, from a concurrency standpoint when building locally with PARALLEL_MAKE=max make -jX all-supported you will face issues with native which are only built once for all archs. That is why it's being done first before invoking parallel make.

[th0ma7]

EDIT2: But it may well be available in a linux compatible binary form from the llvm download page https://releases.llvm.org/download.html

the only available prebuilt version for x64 (amd64) is for freebsd11 https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/clang+llvm-9.0.1-amd64-unknown-freebsd11.tar.xz
But this archive is missing clang-tblgen too.
What about self hosting the native/llvm-9, once built locally?

@th0ma7 I started a test to try selfhosted prebuilt native llvm-9.0 (local builds have a slightly different build path (/spksrc/native/llvm-9.0/ vs /github/workspace/native/llvm-9.0/) but library dependencies are relative, so this should not be a problem. 🤞

The prebuilt package is only 55MB tar.gz, so this will be a fast install of native/llvm-9.0.

Could we self-host it on synocommunity.com directly? Another option could be https://rpmfind.net/linux/rpm2html/search.php?query=clang-tblgen&submit=Search+...&system=&arch=x86_64

EDIT: I actually wonder are both tlbgen mandatory or just llvm (and clang could be discarded)? Testing currently...

EDIT2: I've created a testing PR for clang-tlbgen specifically #5885. I recall at least two out of the 3 native tools where mandatory but I had added them all as they where options available and built anyway.

EDIT3: clang-tlbgen is indeed needed, see build output of #5885.

[th0ma7]

@hgy59 can you merge to master 19b2a78 if it does fix builds? I'll rebase #5879 accordingly.

[hgy59]

@hgy59 can you merge to master 19b2a78 if it does fix builds? I'll rebase #5879 accordingly.

done

[hgy59]

You are right in a github-action environment... But when building locally using make all-supported it's totally different and the concurrency problem hits you real hard. For that the download problem was fixed using flock. On github-action as it spawn multiple VMs we pre-download and mount for each VM afterwards to avoid duplicating data (i.e. different use-case).

with this in mind, the pre-build-native target should only be used in the context of make all-supported.

BTW I never use make all-supported, since I always want to build all and not terminate on the first error.

Anyway, with the use of prebuilt native/llvm-9 this is not a priority issue anymore.

[th0ma7]

You are right in a github-action environment... But when building locally using make all-supported it's totally different and the concurrency problem hits you real hard. For that the download problem was fixed using flock. On github-action as it spawn multiple VMs we pre-download and mount for each VM afterwards to avoid duplicating data (i.e. different use-case).

with this in mind, the pre-build-native target should only be used in the context of make all-supported.

Probably right indeed.

BTW I never use make all-supported, since I always want to build all and not terminate on the first error.

Well it the arch that breaks stops, remaining parallel jobs continues (i.e. PARALLEL_MAKE=max make -j4 all-supported). Note that you absolutely need to use both PARALLEL_MAKE and -jX flag.

Anyway, with the use of prebuilt native/llvm-9 this is not a priority issue anymore.

Even better! Still, I'll check later on if I can enforce building native only from all-supported.

EDIT: It is enforced for only all-supported variants:

$(PUBLISH)all-$(ACTION): | pre-build-native

[th0ma7]

LGTM