参数列表过长，无法通过bpf syscall加载eBPF程序时

Question

I am trying to load an eBPF program via the bpf syscall in Go but am seeing an error returned from the syscall. In order to restrict the problem I am using the following minimal eBPF program, which does nothing:

struct task_group {};    

The important parts of the Go program are as follows:

b, err := ioutil.ReadFile("bpf/bbf_tty.o")
if err != nil {
    fmt.Print(err)
}

progType := BPF_PROG_TYPE_KPROBE
insns := unsafe.Pointer(&b)
insnCnt := len(b)

lba := struct {
    progType    uint32
    pad0        [4]byte
    insnCnt     uint32
    pad1        [4]byte
    insns       uint64
    license     uint64
    logLevel    uint32
    pad2        [4]byte
    logSize     uint32
    pad3        [4]byte
    logBuf      uint64
    kernVersion uint32
    pad4        [4]byte
}{
    progType:    uint32(progType),
    insns:       uint64(uintptr(insns)),
    insnCnt:     uint32(insnCnt),
    license:     uint64(uintptr(0)),
    logBuf:      uint64(uintptr(0)),
    logSize:     uint32(0),
    logLevel:    uint32(0),
    kernVersion: uint32(4),
}

ret, _, err := unix.Syscall(
    unix.SYS_BPF,
    bpf.BPF_PROG_LOAD,
    uintptr(unsafe.Pointer(&lba)),
    unsafe.Sizeof(lba),
)

if ret != 0 || err != 0 {
    return fmt.Errorf("Unable to load program: %s", err)
}

However the error that's getting returned is Unable to load program: argument list too long. Why is this? Or better yet, how can I get a more verbose output to find out the root cause of the issue?

From here there are only three places that E2BIG (argument list too long) gets returned from the bpf syscall, but none of them seem to fit.

I can provide a more complete version of my code if needed, I just tried to strip out the irrelevant parts for brevity.

To help with recreating this issue, I have included my full BPF program below. The full repo is here:

#include <node_config.h>
#include <netdev_config.h>
#include <filter_config.h>

#include <bpf/api.h>

#include <stdint.h>
#include <stdio.h>

#include <linux/bpf.h>
#include <linux/if_ether.h>

#include "lib/utils.h"
#include "lib/common.h"
#include "lib/maps.h"
#include "lib/xdp.h"
#include "lib/eps.h"
#include "lib/events.h"

// define structures
enum pid_type
{
    PIDTYPE_PID,
    PIDTYPE_PGID,
    PIDTYPE_SID,
    PIDTYPE_MAX,
    // only valid to __task_pid_nr_ns() 
    __PIDTYPE_TGID
};
struct upid {
  int nr;
};
struct pid
{
  struct upid numbers[1];
};
struct pid_link
{
  struct pid *pid;
};
struct task_group {
};
struct task_struct {
  struct task_struct *group_leader;
  struct pid_link           pids[PIDTYPE_MAX];
};
struct sid_t {
    int sid;
};

#define BUFSIZE 256
struct tty_write_t {
    int count;
    char buf[BUFSIZE];
    unsigned int sessionid;
};

// define maps
struct bpf_elf_map __section_maps active_sids = {
    .type       = BPF_MAP_TYPE_HASH,
    .size_key   = sizeof(struct sid_t),
    .size_value = sizeof(uint64_t),
};

struct bpf_elf_map __section_maps tty_writes = {
    .type       = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
};

// save_sid saves a sessionid generated from a call
// to setsid to the active_sids map
int save_sid(struct pt_regs *ctx) {

    struct sid_t sid_struct = {};
    int sid = PT_REGS_RC(ctx);
    uint64_t time_ns = bpf_ktime_get_ns();

    sid_struct.sid = sid;

    bpf_map_update(&sid_struct, &time_ns);

    return 0;

}

//int kprobe__tty_write(struct pt_regs *ctx, struct file *file, const char __user *buf, size_t count)
int kprobe__tty_write(struct pt_regs *ctx, struct file *file, const char *buf, size_t count)
{
    struct task_struct *task;
    struct pid_link pid_link;
    struct pid pid;
    int sessionid;

    // get current sessionid
    task = (struct task_struct *)bpf_get_current_task();
    bpf_probe_read(&pid_link, sizeof(pid_link), (void *)&task->group_leader->pids[PIDTYPE_SID]);
    bpf_probe_read(&pid, sizeof(pid), (void *)pid_link.pid);
    sessionid = pid.numbers[0].nr;

    // build session struct key
    struct sid_t sid_key;
    sid_key.sid = sessionid;

    // if sid does not exist in our map then return
    //u64 *time_ns = active_sids.lookup(&sid_key);
    //if (!time_ns) {
    //    return 0;
    //}

    // bpf_probe_read() can only use a fixed size, so truncate to count
    // in user space:
    struct tty_write_t tty_write = {};
    bpf_probe_read(&tty_write.buf, BUFSIZE, (void *)buf);
    if (count > BUFSIZE) {
        tty_write.count = BUFSIZE;
    } else {
        tty_write.count = count;
    }

    // add sessionid to tty_write structure and submit
    tty_write.sessionid = sessionid;
    bpf_perf_event_output(ctx, &tty_write, sizeof(tty_write));

    return 0;
}

Answer 1

Your problem here is the way you try to load the BPF bytecode.

b, err := ioutil.ReadFile("bpf/bbf_tty.o")

I have never used Go, but from what I understand this reads all the bytes from the ELF object file, without any specific processing, and feed them to the bpf() syscall later in your code.

The thing is, this is not how things work: when it compiles into eBPF, clang puts your program into one particular section (by default, .text, but you could specify another name). In addition, if you use eBPF maps, some magic happens (“map relocation”) so that your ELF file can embed map info, and your userspace program calling to bpf() can retrieve it and send it to the kernel.

So when you load the whole file to send it to bpf(), you load your actual bytecode, plus all ELF sections and header. The kernel probably does not like it much. I don't know how to fix it in Go, but here are some pointers that might be helpful:

• libbpf, a C library that can load eBPF programs from the ELF files: located in the kernel tree.
• Gobpf, some framework to use eBPF programs with Go (link). I've never used it, but surely they would have some code to load programs from object files?