Q: Attach xdp program to interface raises "cannot allocate memory"

[csulrong]

Run the minimum example in readme https://pkg.go.dev/github.com/asavie/xdp#section-readme, get "cannot allocate memeory" panic at program.Attach.

	if err := program.Attach(link.Attrs().Index); err != nil {
		panic(err)
	}

Here is my Ubuntu kernel version:

root@emu:/home/emu# uname -a
Linux emu 5.4.0-62-generic #70-Ubuntu SMP Tue Jan 12 12:45:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

And already increased locked memory

root@emu:/home/emu# ulimit -l
4194304

[csulrong]

@slavc I am eager to use this package to boot the performance of packet rx / tx for my application. Do you have any idea on the potential issue here?

[slavc]

Hm, I can't seem to reproduce with the minimal example nor with e.g. rebroadcast example in my environment (QEMU/KVM virtual machine, virtio NIC model), see typescript below.

Can you share more details about your environment, like what if any virtual machine are you running in with what NIC driver, are you running the program as root, etc?

root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# git show -s --oneline HEAD
711132c (HEAD -> master, origin/master, origin/HEAD) Add LoadProgram(): loads an external BPF kernel program (.o file) along with the maps in it (#14)
root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# git diff .
diff --git a/examples/rebroadcast/rebroadcast.go b/examples/rebroadcast/rebroadcast.go
index f37ed97..4fb15e4 100644
--- a/examples/rebroadcast/rebroadcast.go
+++ b/examples/rebroadcast/rebroadcast.go
@@ -88,7 +88,7 @@ func main() {
                        // descriptors and push them onto the Fill ring queue
                        // for the kernel to fill them with the received
                        // frames.
-                       xsk.Fill(xsk.GetDescs(n))
+                       xsk.Fill(xsk.GetDescs(n, true))
                }
                fmt.Printf(" fill=%d, rx=%d, tx=%d, complete=%d\n", xsk.NumFilled(), xsk.NumReceived(), xsk.NumTransmitted(), xsk.NumCompleted())

root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# virt-what
kvm
root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# uname -a
Linux u005 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# ulimit -l
65536
root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# ethtool -i enp8s0
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:08:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# ethtool -l enp8s0
Channel parameters for enp8s0:
Pre-set maximums:
RX:             0
TX:             0
Other:          0
Combined:       1
Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       1

root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# ./rebroadcast -linkname enp8s0
>>> ITERATION 0 <<<
 fill=64, rx=0, tx=0, complete=0
waiting for events...
 fill=64, rx=1, tx=0, complete=0
received: 1
completed: 0
received frame 0:
hexdump:
00000000  01 80 c2 00 00 00 fe 54  00 8e 5e 7a 00 26 42 42  |.......T..^z.&BB|
00000010  03 00 00 00 00 00 80 00  52 54 00 38 b0 06 00 00  |........RT.8....|
00000020  00 00 80 00 52 54 00 38  b0 06 80 04 00 00 14 00  |....RT.8........|
00000030  02 00 02 00                                       |....|


PACKET: 52 bytes
- Layer 1 (14 bytes) = Ethernet {Contents=[..14..] Payload=[..38..] SrcMAC=fe:54:00:8e:5e:7a DstMAC=01:80:c2:00:00:00 EthernetType=LLC Length=38}
- Layer 2 (03 bytes) = LLC      {Contents=[66, 66, 3] Payload=[..35..] DSAP=66 IG=false SSAP=66 CR=false Control=3}
- Layer 3 (35 bytes) = STP      {Contents=[..35..] Payload=[]}


sent: 1
 fill=63, rx=0, tx=1, complete=0
>>> ITERATION 1 <<<
 fill=63, rx=0, tx=1, complete=0
waiting for events...
 fill=63, rx=0, tx=1, complete=0
received: 0
completed: 0
>>> ITERATION 2 <<<
 fill=63, rx=0, tx=1, complete=1
waiting for events...
 fill=63, rx=0, tx=0, complete=0
received: 0
completed: 1
>>> ITERATION 3 <<<
 fill=63, rx=0, tx=0, complete=0
waiting for events...
 fill=63, rx=1, tx=0, complete=0
received: 1
completed: 0
received frame 0:
hexdump:
00000000  01 80 c2 00 00 00 fe 54  00 8e 5e 7a 00 26 42 42  |.......T..^z.&BB|
00000010  03 00 00 00 00 00 80 00  52 54 00 38 b0 06 00 00  |........RT.8....|
00000020  00 00 80 00 52 54 00 38  b0 06 80 04 00 00 14 00  |....RT.8........|
00000030  02 00 02 00                                       |....|


PACKET: 52 bytes
- Layer 1 (14 bytes) = Ethernet {Contents=[..14..] Payload=[..38..] SrcMAC=fe:54:00:8e:5e:7a DstMAC=01:80:c2:00:00:00 EthernetType=LLC Length=38}
- Layer 2 (03 bytes) = LLC      {Contents=[66, 66, 3] Payload=[..35..] DSAP=66 IG=false SSAP=66 CR=false Control=3}
- Layer 3 (35 bytes) = STP      {Contents=[..35..] Payload=[]}


sent: 1
 fill=62, rx=0, tx=1, complete=0
>>> ITERATION 4 <<<
 fill=62, rx=0, tx=1, complete=1
waiting for events...
 fill=62, rx=0, tx=0, complete=0
received: 0
completed: 1
>>> ITERATION 5 <<<
 fill=62, rx=0, tx=0, complete=0
waiting for events...
^C
root@u005:/home/schagaev/go/src/github.com/asavie/xdp/examples/rebroadcast# 

[csulrong]

@slavc I have re-created my VM, and the program works on my new VM now. The difference between my destroyed VM and current in-use VM is the number of queues on the network interfaces. The destroyed VM has 4 queues, and now it's 1. Not sure if this will be the potential issue.

Here is the hardware settings on my current VM:

emu@emu:~$ ethtool -l ens3
Channel parameters for ens3:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	1
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1

[csulrong]

I reproduced the same issue on another new VM.

root@emu:/home/emu# ./rebroadcast -linkname ens5
error: failed to attach xdp program to interface: cannot allocate memory

However, when I explicitly set the XDP flag use generic XDP, it's working.

xdp.DefaultXdpFlags = unix.XDP_FLAGS_SKB_MODE

It seems my virtio driver doesn't support native XDP?
@slavc Do you know any method to check if the native XDP mode is supported or not? If possible, any programmatic way?

root@emu:/home/emu# ethtool -i ens4
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:04.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

root@emu:/home/emu# ethtool -l ens4
Channel parameters for ens4:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	1
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	1