[Help] Same eBPF program output wrong values when loading from Go (compared to libbpf with C++)

[linhanphan]

Hello everyone, I am developing an eBPF application and having a very strange issue. I hope you could give me some clues to fix it.

My usespace application is writing in Go (using cilium/ebpf package). I use the eBPF program from below example:
https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcptracer.bpf.c

I only use one hook point as SEC("kretprobe/inet_csk_accept") to get socket information of a new TCP accept event. I got weird value of IP address and port (no proglem with C++ userspace app). After several days try to clear any differences in environment, I still cannot find the root cause.

Below is my settings to compare between Go and C++

• Use one eBPF program (writing in CO-RE style like the link), compile this file to ELF (.o) file.

• Both Go app and C++ app read the ELF file and load to kernel (Ubuntu 22.04, kernel 6.5). The C++ is from GitHub link, the Go is similar to the guideline of this project (ebpf.LoadCollectionSpec("bpf_prog.o"))

• Both program hook to the same function (inet_csk_accept), same machine, same time. There are two logs from kernel for each event (like below)

   <...>-735803  [007] ...21 2411299.748243: bpf_trace_printk: Listen TCP (from C), family: 10, sport: 80, dport: 61114
   <...>-735803  [007] ...21 2411299.748244: bpf_trace_printk: sock len new: 760
  
   <...>-735803  [007] ...21 2411299.748255: bpf_trace_printk: Listen TCP (from Go), family: 21825, sport: 65161, dport: 22089
   <...>-735803  [007] ...21 2411299.748255: bpf_trace_printk: sock len new: 760
  

As suggested from ChatGPT, I also dump the loaded eBPF program (bpftool prog dump xlated ID) and compare the outputs, they are same.

I am thinking to get pointer address of returned socket, but logically it seems the hook function is called one, then two eBPF programs are called, at the same time, for the same event but the output (from printk) are different. I have no idea what is the next debug approach. Any suggestions or directions are appreciate. Thank you everyone.

[lmb]

It's possible that something is going wrong with the CO-RE relocation. Can you provide the output of bpftool prog dump xlated for both programs? And maybe put the Go code in a repository somewhere?

[linhanphan]

bcc_dump.txt
go_dump.txt

Compare these file, only few lines are different due to map id

I print base pointer, instruction pointer of pt_regs struct (I am not sure the way I print is corrected, but there is different in instruction pointer)

ptr = PT_REGS_FP(ctx);
bp = BPF_CORE_READ(ctx, bp);
ip = PT_REGS_IP(ctx);

// I cannot print these value directly so assign them to fields of struct then print the struct
struct tcp_event e = {};
e.bp = bp;

I cannot upload full code but the load ELF like the guideline:

colSpec, err := ebpf.LoadCollectionSpec(file_path)
col, err := ebpf.NewCollectionWithOptions(colSpec, ebpf.CollectionOptions{
})
for _, programSpec := range colSpec.Programs {
// Attach program
}

I also realize that the eBPF program that is loaded by Go always print the same values of socket ( AF family, sport, dport) regardless of connections (sport, dport should be varied)

[lmb]

Are you on the latest version of the library? How are you attaching the programs?

[linhanphan]

Yes, I am using latest version of this package.

Below is how I attach the program.

	colSpec, err := ebpf.LoadCollectionSpec("ebpf.o")
	if err != nil {
		return fmt.Errorf("failed to load collection spec: %w", err)
	}
	col, err := ebpf.NewCollectionWithOptions(colSpec, ebpf.CollectionOptions{
		Programs: ebpf.ProgramOptions{LogLevel: 1, LogSize: 20 * 1024 * 1024},
	})
	if err != nil {
		return fmt.Errorf("failed to load collection: %w", err)
	}
	for _, programSpec := range colSpec.Programs {
		program := col.Programs[programSpec.Name]
		var l link.Link
		switch programSpec.Type {
		case ebpf.TracePoint:
			parts := strings.SplitN(programSpec.AttachTo, "/", 2)
			l, err = link.Tracepoint(parts[0], parts[1], program, nil)
		case ebpf.Kprobe:
			l, err = link.Kprobe(programSpec.AttachTo, program, nil)
		}
		if err != nil {
			t.Close()
			return fmt.Errorf("failed to link program: %w", err)
		}
	}

I used bpftool to check the program was load successfully, then I got the log as my post. I also use bpftool to load the ELF file (.o) manually and it work well (corrected logs). I guess the difference here is the loader (but at the end, the byte code still are the same).

[lmb]

Could it be that you are attaching the kretprobe via Kprobe instead of https://pkg.go.dev/github.com/cilium/ebpf/link#Kretprobe?

[linhanphan]

Change to Kretprobe has solved problem. Thank you for your suggestion. I did not check it have separated function.