Seeking input regarding fix for indexer segfault, and shared object search paths

[johnny-mayo]

I've figured out exactly what the problem with indexer is and how to fix it.
There must be better solutions...please let me know!

In a nutshell, the .interp attribute needs to be set, something like:

gcc -o myapp main.o -Wl,--dynamic-linker=ld-linux.so.2

...if gcc will let you...very much against the rules to specify the interpreter without an absolute path to it.

Here is a demonstration:

(base) user@erying:~/gittyup/appimage$ ./Gittyup-1.4.0-x86_64.AppImage --appimage-extract
# ...files are extracted to squashfs-root
(base) user@erying:~/gittyup/appimage$ cd squashfs-root/usr/bin/
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./indexer
Segmentation fault

# Let's see what the .interp attribute is currently set to:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ readelf -a ./indexer | grep interpreter
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

# This is bad. It loads ld-linux from the host system instead of the one it is packaged with
# On some systems, the host ld-linux won't be compatible with the programs/libraries in an AppImage,
# ...and "pop goes the weasel"

# Let's just fix that:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter ld-linux-x86-64.so.2 indexer

# Yikes!...? We are setting the interpreter without an absolute path to it, clearly against all sanity!
# (more on this below)

# Does it run?
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./indexer
Usage: ./indexer [options] repo

Options:
  -h, --help        Displays help on commandline options.
  --help-all        Displays help including Qt specific options.
  -l, --log         Write indexer progress to log.
  -v, --verbose     Print indexer progress to stdout.
  -n, --notify      Notify when data is written to disk.
  -b, --background  Start with background priority.

Arguments:
  repo              path to repository

# Seems to work just fine. Now that we've kicked the tires, let's take it for a spin.
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ cd ../..
(base) user@erying:~/gittyup/appimage/squashfs-root$ ./AppRun

# While you are in Gittyup, initialize a new repository or open an existing one
# Result: works just fine. No indexer error. Check the logs, there is nothing. Everything "just works".

Discussion:

Q: Why does indexer need the .interp attribute changed and not gittyup?
A: Because gittyup is run by the correct ld-linux in Apprun

Q: If we run gittyup with the correct ld-linux, they why do we have to "fix" indexer?
A: Because, when indexer is run by gittyup, we are not specifying the ld-linux to run it with, so indexer will
    use what is currently in its .interp attribute, which will load it from the host, and it if isn't compatible
    ...yea, we talked about that already...

Q: Setting the .interp attribute without an absolute path to it is EVIL! (in the form of a question)
A: The rules and reasons for .interp, RUNPATH and other various attributes exist for normal runtime conditions,
    but are irrelevant and/or antithetical to working in an isolated and portable environment, like an AppImage,
    in this case.

Q: How do you know it will work on every system?
A: Because we control the ld-linux being used. Since it works on one system it should work on all systems.

Also consider: there are other programs in squashfs-root/usr/bin that may require the same treatment. I don't know if
there are reports of their failure when running Gittyup from an AppImage, but if there are, this may fix it.

# List other executables:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ find ../.. -type f -exec file {} + | grep 'ELF.*executable' | cut -d: -f1
../../usr/bin/cmark
../../usr/bin/credential-helpers/git-credential-libsecret
../../usr/bin/gittyup
../../usr/bin/indexer
../../usr/bin/relauncher

# Hmm, let's see:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./cmark
*** stack smashing detected ***: terminated
Aborted
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./relauncher
Segmentation fault
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./gittyup
Segmentation fault
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ credential-helpers/git-credential-libsecret 
Segmentation fault

# OK, let's try "fixing" them:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter ld-linux-x86-64.so.2 cmark
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter ld-linux-x86-64.so.2 relauncher
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter ld-linux-x86-64.so.2 gittyup
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter ld-linux-x86-64.so.2 credential-helpers/git-credential-libsecret
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ cp ld-linux-x86-64.so.2 credential-helpers/
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./cmark
^C
# cmark doesn't crash and was listening to stdin, like its man page says it should...
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./relauncher
# relauncher doesn't crash any more...no idea what it does.
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ credential-helpers/git-credential-libsecret
usage: git-credential-libsecret <get|store|erase>
# git-credential-libsecret doesn't crash any more

# Ooh, let's try running Gittyup WITHOUT AppRun!
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./gittyup
# The GUI comes up, seemingly functional, and when I create or open a project, indexer works fine

# How do I know that Gittyup isn't loading shared objects from the host, without AppRun's help?
ldd ./gittyup | less
# It isn't.

Another thing: The .interp attribute in an ELF executable is relative to the current working directory. This
means that, although the working directory is changed to usr/ in AppRun before gittyup is launched,
either something in gittyup is changing the working directory, or the working directory is defaulting
to the directory of indexer or gittyup...or something else...

# For example, let's break it by specifying a relative path:  bin/
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter bin/ld-linux-x86-64.so.2 indexer
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./indexer
bash: ./indexer: cannot execute: required file not found

# Let's go to where it will work:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ cd ..
(base) user@erying:~/gittyup/appimage/squashfs-root/usr$ bin/indexer
Usage: bin/indexer [options] repo

Options:
  -h, --help        Displays help on commandline options.
  --help-all        Displays help including Qt specific options.
  -l, --log         Write indexer progress to log.
  -v, --verbose     Print indexer progress to stdout.
  -n, --notify      Notify when data is written to disk.
  -b, --background  Start with background priority.

Arguments:
  repo              path to repository

# Let's go back and fix it again:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr$ cd bin
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-interpreter ld-linux-x86-64.so.2 indexer

Note: $ORIGIN expands to the directory the ELF executable is located in, and can be used in many ELF
attributes, but not .interp. Tried it. See the "Dynamic string tokens" section of "man ld-linux" for more info.

I apologize for not bringing this to the community earlier. I was diving into infinite rabbit holes on ELF executable
shared object loading optimization the last few days. This is the best I can do in that respect, so far:

# ELF executable shared object loading optimization (should work for the shared objects themselves, as well)

# Starting from the same squashfs-root directory that we "fixed" indexer in:
(base) user@erying:~/gittyup/appimage/squashfs-root$ cd usr/bin

# We are going to make a backup copy of indexer we already fixed .interp for:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ cp indexer indexer.org

# Here is the list of shared objects indexer wants...well...NEEDs:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ readelf -all indexer | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libpcre.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libQt5Network.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libQt5Concurrent.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libQt5Core.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

# It will be the same in indexer.org.

# Let's take a look at the unique paths these libraries resolve to currently. Where are the .so files found?
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ldd ./indexer.org | grep ' => ' | grep -v 'ld-linux' | cut -d ' ' -f 3- | cut -d '(' -f 1 | xargs realpath "{}" | sed -n 's|.*/squashfs-root/\(.*\)/.*|\1|p' | sort | uniq
home/runner/work/Gittyup/Qt/5.15.2/gcc_64/lib
lib/x86_64-linux-gnu
usr/bin
usr/lib/x86_64-linux-gnu

# Hmm. So it can find every shared object it needs in only four paths?

# Remember, $ORIGIN resolves to the path the ELF shared object or executable is in, and a relative path can
# be constructed from that point for RPATH, RUNPATH, LD_LIBRARY_PATH...see "man ld-linux" for more info.

# Let's take a look at the shared object search paths in the original indexer. Where is it told to search?:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ readelf -a indexer.org | grep RUNPATH | sed -n 's/.*\[\(.*\)\].*/\1/p' | tr ':' '\n'
$ORIGIN/.
$ORIGIN/credential-helpers
$ORIGIN/../lib/x86_64-linux-gnu
$ORIGIN/../lib64
$ORIGIN/../../lib64
$ORIGIN/../lib
$ORIGIN/../../lib
$ORIGIN/../lib/x86_64-linux-gnu/libfakeroot
$ORIGIN/../local/lib
$ORIGIN/../local/lib/x86_64-linux-gnu
$ORIGIN/../../lib/x86_64-linux-gnu
$ORIGIN/../../lib32
$ORIGIN/../lib32
$ORIGIN/../../opt/hostedtoolcache/Python/3.11.9/x64/lib
$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/lib
$ORIGIN/../../home/qt/icu_install/lib
$ORIGIN/../../lib/x86_64-linux-gnu/gconv
$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/plugins/platforms
$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/plugins/iconengines
$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/plugins/imageformats
$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/plugins/xcbglintegrations
$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/plugins/bearer

# That's a little more than four paths...

# Further, all of these search paths are multiplied by various additional search paths tacked on:
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./ld-linux-x86-64.so.2 --help
...
Shared library search path:
  (libraries located via /EEE/ld.so.cache)
  /XXX/x86_64-linux-gnu (system search path)
  /xxx/XXX/x86_64-linux-gnu (system search path)
  /XXX (system search path)
  /xxx/XXX (system search path)

Subdirectories of glibc-hwcaps directories, in priority order:
  x86-64-v4 (supported, searched)
  x86-64-v3 (supported, searched)
  x86-64-v2 (supported, searched)

Legacy HWCAP subdirectories under library search path directories:
  haswell (AT_PLATFORM; supported, searched)
  tls (supported, searched)
  avx512_1 (supported, searched)
  x86_64 (supported, searched)

# Look at the bottom two sections of the output. For each path listed in RPATH, RUNPATH, etc.,
# these additional paths are added and searched through.

# For example, let's start at the top with '$ORIGIN/.' This becomes:
$ORIGIN/./glibc-hwcaps/x86-64-v4/
$ORIGIN/./glibc-hwcaps/x86-64-v3/
$ORIGIN/./glibc-hwcaps/x86-64-v2/
$ORIGIN/./tls/haswell/avx512_1/
$ORIGIN/./tls/haswell/avx512_1/
$ORIGIN/./tls/haswell/x86_64/
$ORIGIN/./tls/haswell/
$ORIGIN/./tls/avx512_1/x86_64/
$ORIGIN/./tls/avx512_1/
$ORIGIN/./tls/x86_64/
$ORIGIN/./tls/
$ORIGIN/./haswell/avx512_1/x86_64/
$ORIGIN/./haswell/avx512_1/
$ORIGIN/./haswell/x86_64/
$ORIGIN/./haswell/
$ORIGIN/./avx512_1/x86_64/
$ORIGIN/./avx512_1/

# ...and then the next line, '$ORIGIN/credential-helpers' is multiplied the same way
# ...and so on, and so on...

# This is where glibc "prefers" shared objects that are optimized for accelerated processor extensions
# ...if they exist...but they don't here in our AppImage environment.

If you want to see all the searching madness, you can:

# Use LD_DEBUG=all
LD_DEBUG=all ./indexer.org |& less

# Use strace:
strace --follow-forks ./indexer.org |& less

# Let's set RUNPATH so it only looks in the four paths where it will find its libraries, instead of
# the 22 paths it was looking through. Note: These paths will still be multiplied as mentioned above
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ patchelf --set-rpath '$ORIGIN/../lib/x86_64-linux-gnu:$ORIGIN/../../lib/x86_64-linux-gnu:$ORIGIN/../../home/runner/work/Gittyup/Qt/5.15.2/gcc_64/lib' indexer

# Now, lets compare the before and after in the number of paths searched for each library indexer NEEDs.

# For indexer.org (before):
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ LD_DEBUG=all ./indexer.org |& awk '/file=.*needed by \.\/indexer/ {flag=1; file_line=$0; count=0} /trying file=/ && flag {count++} /file=.*generating link map/ && flag {print file_line, "Count:", count; flag=0}'
     16045:	file=libpcre.so.3 [0];  needed by ./indexer.org [0] Count: 209
     16045:	file=libz.so.1 [0];  needed by ./indexer.org [0] Count: 8
     16045:	file=libssl.so.1.1 [0];  needed by ./indexer.org [0] Count: 3
     16045:	file=libcrypto.so.1.1 [0];  needed by ./indexer.org [0] Count: 3
     16045:	file=libQt5Network.so.5 [0];  needed by ./indexer.org [0] Count: 84
     16045:	file=libQt5Concurrent.so.5 [0];  needed by ./indexer.org [0] Count: 9
     16045:	file=libQt5Core.so.5 [0];  needed by ./indexer.org [0] Count: 9
     16045:	file=libstdc++.so.6 [0];  needed by ./indexer.org [0] Count: 8
     16045:	file=libm.so.6 [0];  needed by ./indexer.org [0] Count: 8
     16045:	file=libgcc_s.so.1 [0];  needed by ./indexer.org [0] Count: 8
     16045:	file=libc.so.6 [0];  needed by ./indexer.org [0] Count: 8

# For indexer (after):
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ LD_DEBUG=all ./indexer |& awk '/file=.*needed by \.\/indexer/ {flag=1; file_line=$0; count=0} /trying file=/ && flag {count++} /file=.*generating link map/ && flag {print file_line, "Count:", count; flag=0}'
     16073:	file=libpcre.so.3 [0];  needed by ./indexer [0] Count: 38
     16073:	file=libz.so.1 [0];  needed by ./indexer [0] Count: 2
     16073:	file=libssl.so.1.1 [0];  needed by ./indexer [0] Count: 1
     16073:	file=libcrypto.so.1.1 [0];  needed by ./indexer [0] Count: 1
     16073:	file=libQt5Network.so.5 [0];  needed by ./indexer [0] Count: 21
     16073:	file=libQt5Concurrent.so.5 [0];  needed by ./indexer [0] Count: 3
     16073:	file=libQt5Core.so.5 [0];  needed by ./indexer [0] Count: 3
     16073:	file=libstdc++.so.6 [0];  needed by ./indexer [0] Count: 2
     16073:	file=libm.so.6 [0];  needed by ./indexer [0] Count: 2
     16073:	file=libgcc_s.so.1 [0];  needed by ./indexer [0] Count: 2
     16073:	file=libc.so.6 [0];  needed by ./indexer [0] Count: 2

# Each shared object needed by indexer has a "Count: " at the end of the line for the number of paths that
# were searched, the last path of which finally found that particular shared object.

# Ok...does it still run?
(base) user@erying:~/gittyup/appimage/squashfs-root/usr/bin$ ./indexer
Usage: ./indexer [options] repo

Options:
  -h, --help        Displays help on commandline options.
  --help-all        Displays help including Qt specific options.
  -l, --log         Write indexer progress to log.
  -v, --verbose     Print indexer progress to stdout.
  -n, --notify      Notify when data is written to disk.
  -b, --background  Start with background priority.

Arguments:
  repo              path to repository


Discussion:

Q: Do you really expect us to hack/patch/modify the binary with patchelf?
A: No. Something like "gcc -o indexer main.o -Wl,-runpath$ORIGIN/../path/to/libs:$ORIGIN/../../path/to/more/libs"
should be done in the AppImage build.

Q: Wait, if you are doing a --set-rpath, why is patchelf changing the RUNPATH?
A: Read "man patchelf" for details

Q: Why don't you use "patchelf --shrink-path"? It is supposed to do exactly what you are doing manually!
A: I tried that. It didn't work for me.


# Some other stuff I've messed with:

# Setting the SONAME to an $ORIGIN relative path to the .so
patchelf --set-soname ...

# Adding/modifying the NEEDED to specify an $ORIGIN relative path to the .so file
patchelf --add-needed ...

I think one of these worked when I changed only one item, but neither will work if I change all of them.
Something is sensitive to this. Dunno. It would take a lot more probing to figure it out.

# Setting RPATH instead of RUNPATH will assure that what is found in RPATH will not be sought
# in LD_LIBRARY_PATH, which would help keep a (mis)configuration with LD_LIBRARY_PATH set
# from throwing a monkey wrench into the works.

# LD_LIBRARY_PATH will be sought before RUNPATH (but after RPATH), but is not normally advisable.

# Using LD_PRELOAD might be an idea for libraries common to all, but is normally not advisable.

# "normally"

# I tried several things to eliminate the additional search paths for optimized .so files that
# don't exist. Since it is a moving target from one glibc version to another and one distro to
# another, I tried several variations of:
export GLIBC_TUNABLES=glibc.cpu.hwcap_mask=0x0
...and...
"./ld-linux-x86-64.so.2 --glibc-hwcaps-mask LIST my_program"
# ...but no dice...or, maybe it does work to turn off the cpu capabilities, but the paths are
# uselessly searched nonetheless...or maybe I just bungled something.


# Running "./ld-linux-x86-64.so.2 --help" is recommended for anyone interested. Some interesting options are:

--list                       list all dependencies and how they are resolved
--verify                     verify that given object really is a dynamically linked object we can handle
--inhibit-cache              Do not use /EEE/ld.so.cache
--library-path PATH          use given PATH instead of content of the environment variable LD_LIBRARY_PATH
--glibc-hwcaps-prepend LIST  search glibc-hwcaps subdirectories in LIST
--glibc-hwcaps-mask LIST     only search built-in subdirectories if in LIST
--inhibit-rpath LIST         ignore RUNPATH and RPATH information in object names in LIST
--audit LIST                 use objects named in LIST as auditors
--preload LIST               preload objects named in LIST
--argv0 STRING               set argv[0] to STRING before running
--list-tunables              list all tunables with minimum and maximum values
--list-diagnostics           list diagnostics information


<tangent>

ldd will only tell us how the libraries would resolve to .so files, but does not consider what happens
later in the process of an ELF loading, determining if the resolved .so files contain the required symbols,
_and_ the correct symbol versions.

Normally, when a program starts, it will first find the NEEDED .so files, but function call resolution does
not occur until those functions are first referenced, and, at the point of function resolution, it is determined
whether the correct symbols/versions exist in the found .so files, and if they don't, the whole thing blows
up instead of continuing the search for the correct symbols/versions. This behavior can be modified by the
LD_BIND_NOW (since glibc 2.1.1) environment variable, which, when set to any non-empty string, makes
symbol/version resolution happen "at program startup", instead of when they are first referenced (man ld-linux).

Again, if you want to see all the searching madness, you can:

# Use LD_DEBUG=all
LD_DEBUG=all ./indexer.org |& less

# Use strace:
strace --follow-forks ./indexer.org |& less

</tangent>

Anyway, this is all new to me...stuff I figured out the last few days...I'm probably wrong about more
than one thing, there are probably solutions for some of these problems I don't know about, and I
would love to know what what they are. Maybe I'm using the wrong google searches?

 I've accumulated a lot more information, but it is not tested, so...maybe later.

Thank you,

johnny-mayo

P.S. I am learning to dislike markdown...and no line-wrapping without it? Sheesh.

[cakemonitor]

Thank you @johnny-mayo for your work identifying the problem and finding a workaround!

For anyone who wants a step-by-step guide to implementing the proposed fix:

• In a terminal, navigate to the directory of your Gittyup appimage (or copy it to a temporary working folder if you prefer)
• Run the command ./Gittyup-1.4.0-x86_64.AppImage --appimage-extract (replace the version number to match you own if required). This will generate a squashfs-root directory with the extracted files.
• Confirm the indexer fault:
  • cd squashfs-root/usr/bin/.
  • Run ./indexer and verify the output is a segmentation fault.
• If necessary, run sudo apt install patchelf (or equivalent install steps for your system).
• Run patchelf --set-interpreter ld-linux-x86-64.so.2 indexer.
• Rerun ./indexer and verify that it now produces valid output with usage options.
• Return to the appimage directory (cd ../../..).
• Create a new appimage from the modified squashfs-root directory:
  • Download appimagetool-x86_64.AppImage from https://github.com/AppImage/AppImageKit/releases.
    • Move it to your working directory so it's alongside the extracted squashfs-root directory.
    • Make it executable (chmod +x appimagetool-x86_64.AppImage).
  • Try to create your patched appimage by running ARCH=x86_64 ./appimagetool-x86_64.AppImage squashfs-root.
  • If it fails because it cannot find gittyup.png:
    • Create a symlink: ln -s "$(realpath squashfs-root/usr/share/icons/hicolor/256x256/apps/gittyup.png)" squashfs-root/gittyup.png.
    • Rerun ARCH=x86_64 ./appimagetool-x86_64.AppImage squashfs-root.
  • If required, rename your new appimage, e.g. mv Gittyup-x86_64.AppImage Gittyup-1.4.0-patched-indexer-x86_64.AppImage.
• If desired, you can safely delete the original Gittyup appimage, the squashfs-root directory, and the appimagetool appimage.