How to run ecapture in Android 11/12 with kernel 5.4 aarch64: detect and rebuild kernel step

[HorseLuke]

In issue #296 , I have phone with kernel 5.4 aarch64, but ecapture can not success run. After discussed with author, I went to try patch kernel, to see if it can be run on patched kernel 5.4.

Now, with a lot of effort, I managed to run ecapture on android 11/12 with kernel 5.4 aarch64. Here is a brief step, build environment is debian 11 with docker.

Tested on launch kernel android11-5.4 (boot.img) with Android 12 userspace (about page that user can see).

Much thanks from @cfc4n help.

[Easy version]

Buy a new phone with kernel 5.10 or higher. :)

[Normal version]

1. Grab kernel 5.4 source codes
   1.1. If you can, Grab kernel 5.4 source codes from mobile manufacturer.
   1.2 If not, detect whether mobile is compatible with GKI 1.0 [2]. If so, grab kernel 5.4 source codes from aosp.

   (A) Grab stock boot.img.

   (B) use "tools/mkbootimg/unpack_bootimg.py --boot_img=boot.img"[1] to detect os_version. 11 means android11-5.4 branch, 12 means android12-5.4 branch.

   (C) Download the GKI boot image from ci.android.com. android11-5.4 branch use https://ci.android.com/builds/branches/aosp-android11-gsi/grid , android12-5.4 branch is https://ci.android.com/builds/branches/aosp-android12-gsi/grid . Download aosp_arm64-img-xxxx.zip from aosp_arm64 target, and extract to pick file boot-5.4.img

   (D) boot mobile into fastboot mode, connect to pc, then run "fastboot.exe boot boot-5.4.img". If success, it will boot successfully, and kernel version will change in about setting; If not, it will boot failed, or kernel version remains unchanged in about setting, you have to buy a new phone with kernel 5.10 or higher :( .

   (E) If success in D, go to grab kernel 5.4 source codes from aosp, android11-5.4 branch use common-android11-5.4 , android12-5.4 branch use common-android12-5.4 branch[3]

2. Patch probe_read_{user,kernel}() and other codes from kernel upstream

   Patch ALL CODES from torvalds/linux@358fdb4
   I have uploaded my modified code in this link: https://github.com/HorseLuke/aosp_android_common_kernels/tree/android-11-5.4-bpf_probe_read_user
   In branch "android-11-5.4-bpf_probe_read_user", you can see files where I modified. This branch is taken from "android11-5.4" branch.
   If you don't want to modify by yourself, you can just pick it and go

3. Rebuild patched kernel and generate boot.img

   CONFIG_DEBUG_INFO_BTF can be Y or not set.
   Tips: Some phone may not boot or bootloop with CONFIG_DEBUG_INFO_BTF=y, guess that rebuilded kernel with BTF info is too large to boot.
   If you use kernel 5.4 source codes from aosp, menuconfig comand is:

BUILD_CONFIG=common/build.config.gki.aarch64 build/config.sh menuconfig

If you use kernel 5.4 source codes from aosp, build comand is:

BUILD_BOOT_IMG=1 \
SKIP_VENDOR_BOOT=1 \
KERNEL_BINARY=Image \
BOOT_IMAGE_HEADER_VERSION=3 \
GKI_RAMDISK_PREBUILT_BINARY=gki-ramdisk.lz4 \
BUILD_CONFIG=common/build.config.gki.aarch64 \
SKIP_MRPROPER=1 \
/bin/bash ./build/build.sh

(remove SKIP_MRPROPER=1 if you want a full build)
(gki-ramdisk.lz4 is unpacked from boot-5.4.img: )

tools/mkbootimg/unpack_bootimg.py --boot_img=boot-5.4-gz.img
mv $KERNEL_ROOT/out/ramdisk gki-ramdisk.lz4

4. Generate BTF
   If CONFIG_DEBUG_INFO_BTF is Y in step 3, ignore this step. If not, do this step
   After rebuild, you will get vmlinux in output dir. use this command to generate btf

cp -a vmlinux vmlinux.btf
pahole -J vmlinux.btf

5. Generate vmlinux.h

Use bpftool to generate vmlinux.h.

It will take very very very very very long time, and file size of vmlinux.h may be very very very very very large.

Run bpftool outside docker container with newest bpftool version may take less time.

CONFIG_DEBUG_INFO_BTF=Y

bpftool btf dump file vmlinux  format c > vmlinux.h

CONFIG_DEBUG_INFO_BTF=N or not set with step 4 finish:

bpftool btf dump file vmlinux.btf  format c > vmlinux.h

How large? see this:

HorseLuke@b4aa2dd

If you don't want to generate vmlinux.h, you can download "vmlinux-for-modified-kernel-5.4-unzip-to-this-folder.zip" from above link, unzipped to get vmlinux.h.

6. Rebuild ecapture with generated vmlinux.h and no core mode.
   You can run rebuild process in docker arm64 container on x64 host[4][5]. This will save lots of time.
   Before run rebuild, modify Makefile, hardcode UNAME_M and UNAME_R like:

UNAME_M := aarch64
UNAME_R := 5.4.226-android11-2

Don't forget to overwrite /kern/bpf/arm64/vmlinux.h with generated vmlinux.h

Rebuild ecapture with command like this

ANDROID=1 \
	KERN_BUILD_PATH=/aosp1/android-kernel/out/android11-5.4/common \
	KERN_SRC_PATH=/aosp1/android-kernel/common \
	make nocore

If you encountering saveMasterSecretBSSL panic, see this discussion and modify code:

#310

Do not forget modify KERN_BUILD_PATH and KERN_SRC_PATH

7. Flash generated boot.img where step 3 build into mobile.

8. Now bin/ecapture can be run on patched kernel 5.4.

---

[1]https://android.googlesource.com/platform/system/tools/mkbootimg/+/master/unpack_bootimg.py
[2] https://source.android.com/docs/core/architecture/kernel/generic-kernel-image
[3] https://source.android.com/docs/setup/build/building-kernels
[4] https://docs.docker.com/build/building/multi-platform/
[5] https://devopstales.github.io/linux/running_and_building_multi_arch_containers/

[HorseLuke]

I have uploaded my modified code in this link: https://github.com/HorseLuke/aosp_android_common_kernels/tree/android-11-5.4-bpf_probe_read_user

In branch "android-11-5.4-bpf_probe_read_user", you can see files where I modified. This branch is taken from "android11-5.4" branch.

If you don't want to modify by yourself, you can just pick it and go

[HorseLuke]

About generating vmlinux.h

If your dwarves / pahole version is equals to or higher than v1.24, you can use new --compile argument to generate vmlinux.h directly. It will take much much shorter time, and file size is much more smaller than bpftool that generated.

./pahole  --compile /datadisk/aosp1/android-kernel/out/android11-5.4/dist/vmlinux  \
    > /datadisk/aosp1/android-kernel/out/android11-5.4/dist/vmlinux.h