Enable virt/kvm on the kernel for (at least) firmware-nanopi6 and firmware-nanopi5

[tarastella]

Would it be possible to consider enabling kernel virtualisation for the more powerful devices such as NanoPI R6S and R5S?
Need to enable the following on the kernel:

CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y

Thanks,
Tara

[Joulinar]

we don't maintain own kernel for NanoPi. On R5 und R6 its original vendor kernel.

[tarastella]

Does it mean it's the original FriendlyWRT kernel? That would explain the kernel version.
Thanks a lot.

[Joulinar]

Correct

[MichaIng]

Jep, here are the kernel sources: https://github.com/friendlyarm/kernel-rockchip/tree/nanopi5-v5.10.y_opt
Here the config for NanoPi 6 series: https://github.com/friendlyarm/kernel-rockchip/blob/nanopi5-v5.10.y_opt/arch/arm64/configs/nanopi6_linux_defconfig

Armbian is also working on support for R6S. They use the same kernel sources but own builds with virtualization most likely enabled. Once this is ready, we will switch. If mainline Linux + U-Boot support is there at some point, we may also do own kernel builds, but currently, with Debian Bookworm release next week, time does not permit to look deeper into it.

[tarastella]

I was able to source the config back with "modprobe configs", this is how I added the rdb.ko module.
But virtualisation is a bit more complicated. However, amazing news that someone is working on it.
R6S has sufficient RAM to run a few VMs. We (myself and my partner) are currently working to create a NanoPI cluster as an homelab project.

I understand that resources are limited and there are priorities.
Thank you for your amazing work.

[tarastella]

Just as a further update. I was able to compile the Kernel from the FriendlyARM sources and add virtualisation and the required modules. The page on their website is very helpful. I was able to successfully run a VM with a Ceph volume backend today.
If you need any hint, get in touch.
Thank you very much!

[Ewertton]

Hi @tarastella, I need help to enable kvm on kernel from my NanoPi R5S. You help me with this?

[tarastella]

@Ewertton, I'm assuming you can compile on the same node and I'm assuming you have compiled a kernel.
The FriendlyELEC wiki is somehow well done https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R5C#How_to_Compile
However, it describes how to cross-compile it.

So, install these required packages on DietPI:
pkg-config flex make gcc libc6-dev libncurses-dev bison libssl-dev bc cpio xz-utils python

Clone the kernel:

git clone https://github.com/friendlyarm/kernel-rockchip.git
git checkout nanopi5-v5.10.y_opt # (R5S and R6S share the same tree)

Dump the kernel config with depmod configs and copy the /proc/config.gz that has to be decompressed and copied in the kernel source tree as .config. Then add the following lines in .config:

CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y

CONFIG_VHOST=m
CONFIG_VHOST_MENU=y
CONFIG_VHOST_NET=m
CONFIG_VHOST_VSOCK=m
CONFIG_VHOST_CROSS_ENDIAN_LEGACY=y

Run make menuconfig and check that those options are activated.
Backup the content of /lib/modules/5.10.110
Then compile with the following commands that generate the R5S images:

make nanopi5-images -j2
mkdir -p out-modules && rm -rf out-modules/*
make INSTALL_MOD_PATH="$PWD/out-modules" modules -j2
make INSTALL_MOD_PATH="$PWD/out-modules" modules_install
[ ! -f "$PWD/out-modules/lib/modules/${KERNEL_VER}/modules.dep" ] && depmod -b $PWD/out-modules -E Module.symvers -F System.map -w  5.10.110
cd $PWD/out-modules && find . -name \*.ko | xargs strip --strip-unneeded

Copy over the modules that were stored in "out-modules" into /lib/modules/5.10.110
Flash the new kernel with:

dd if=resource.img of=/dev/mmcblk2p4 bs=1M
dd if=kernel.img of=/dev/mmcblk2p5 bs=1M

Before doing that, make sure that the above partitions are right with parted /dev/mmcblk2 print
Also, crosscheck with FriendlyELEC's wiki.
Hope this helps.

[MichaIng]

Great instructions. When I find time, I'll switch to own kernel builds as well. I hope the bootloader sources generally allow to boot with a single partition system without this dedicated partitions 1-7, but with the kernel image and U-Boot configs just in /boot.

[kinhsman]

Hi @tarastella,
I'm not a day-to-day Linux user and never complied a kernel before. I tried to follow your steps but getting stuck here and not sure how to do this:

Dump the kernel config with depmod configs and copy the /proc/config.gz that has to be decompressed and copied in the kernel source tree as .config. Then add the following lines in .config:
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_VHOST=m
CONFIG_VHOST_MENU=y
CONFIG_VHOST_NET=m
CONFIG_VHOST_VSOCK=m
CONFIG_VHOST_CROSS_ENDIAN_LEGACY=y

root@NanoPi-PVE:~/kernel-rockchip# depmod configs
depmod: ERROR: Bad version passed configs

Any insight is greatly appreciated!

[tarastella]

I suspect that the kernel version you are running is somehow not aligned with the modules you have.
I'm travelling right now with no way of trying this, I am afraid.
If you want, I've a kernel compiled for NanoPI 6S that you can download from here.
https://www.dropbox.com/s/re0dkdhb18gc12p/kernel-R6S-with-KVM-RBD-5.10.110.tar.gz?dl=0

[kinhsman]

With a bit of help from ChatGPT and your compiled kernel, I have successfully enabled KVM on my little device. Thank you very much @tarastella !

[tarastella]

Awesome! YVW!! ❤️