weixin_39605278
weixin_39605278
2021-01-10 04:12

PXE boot kernel panic: unable to mount root fs on unknown-block(2,0)

Follow-up of https://github.com/raspberrypi/firmware/issues/862.

When using an SD card with just BOOTCODE.BIN the client boots into the TFTP process:

Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 available DHCP subnet: 192.168.0.255/255.255.255.0
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 PXE(eth0) b8:27:eb:9d:c1:5a proxy
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 tags: eth0
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 broadcast response
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 sent size:  1 option: 53 message-type  2
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 sent size:  4 option: 54 server-identifier  192.168.0.48
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 sent size: 17 option: 97 client-machine-id  00:5a:c1:9d:57:5a:c1:9d:57:5a:c1:9d:57:5a...
Aug 27 12:55:22 keller dnsmasq-dhcp[27427]: 653460281 sent size: 32 option: 43 vendor-encap  06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
Aug 27 12:55:23 keller dnsmasq-tftp[27427]: file /tftpboot/579dc15a/start.elf not found
Aug 27 12:55:23 keller dnsmasq-tftp[27427]: file /tftpboot/autoboot.txt not found
Aug 27 12:55:23 keller dnsmasq-tftp[27427]: sent /tftpboot/config.txt to 192.168.0.66
Aug 27 12:55:23 keller dnsmasq-tftp[27427]: file /tftpboot/recovery.elf not found
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: sent /tftpboot/start_cd.elf to 192.168.0.66
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: sent /tftpboot/fixup_cd.dat to 192.168.0.66
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: file /tftpboot/recovery.elf not found
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: sent /tftpboot/config.txt to 192.168.0.66
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: file /tftpboot/dt-blob.bin not found
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: file /tftpboot/recovery.elf not found
Aug 27 12:55:24 keller dnsmasq-tftp[27427]: sent /tftpboot/config.txt to 192.168.0.66
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/bootcfg.txt not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: sent /tftpboot/cmdline.txt to 192.168.0.66
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/recovery8.img not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/recovery8-32.img not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/recovery7.img not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/recovery.img not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/kernel8.img not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/kernel8-32.img not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/armstub8.bin not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: error 0 Early terminate received from 192.168.0.66
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: failed sending /tftpboot/kernel7.img to 192.168.0.66
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/armstub8-32.bin not found
Aug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/armstub7.bin not foundAug 27 12:55:25 keller dnsmasq-tftp[27427]: file /tftpboot/armstub.bin not found
Aug 27 12:55:30 keller dnsmasq-tftp[27427]: sent /tftpboot/kernel7.img to 192.168.0.66
Aug 27 12:55:30 keller dnsmasq-tftp[27427]: sent /tftpboot/bcm2710-rpi-3-b.dtb to 192.168.0.66
Aug 27 12:55:30 keller dnsmasq-tftp[27427]: sent /tftpboot/config.txt to 192.168.0.66
Aug 27 12:55:35 keller dnsmasq-dhcp[27427]: 1189977513 available DHCP subnet: 192.168.0.255/255.255.255.0
Aug 27 12:55:37 keller dnsmasq-dhcp[27427]: 1189977513 available DHCP subnet: 192.168.0.255/255.255.255.0
Aug 27 12:55:37 keller rpc.mountd[27691]: authenticated mount request from 192.168.0.66:694 for /nfs/client1 (/nfs/client1)
Aug 27 12:55:42 keller rpc.mountd[27691]: authenticated mount request from 192.168.0.66:769 for /nfs/client1 (/nfs/client1)
Aug 27 12:55:52 keller dhcpcd[1390]: eth0: Router Advertisement from fe80::ca0e:14ff:feaa:856a
Aug 27 12:55:52 keller rpc.mountd[27691]: authenticated mount request from 192.168.0.66:670 for /nfs/client1 (/nfs/client1)
Aug 27 12:56:13 keller rpc.mountd[27691]: authenticated mount request from 192.168.0.66:798 for /nfs/client1 (/nfs/client1)
Aug 27 12:56:43 keller rpc.mountd[27691]: authenticated mount request from 192.168.0.66:1017 for /nfs/client1 (/nfs/client1)
Aug 27 12:57:13 keller rpc.mountd[27691]: authenticated mount request from 192.168.0.66:904 for /nfs/client1 (/nfs/client1)

Having the client connected to a monitor I can see it stalls after IP-Config: Complete, the ip config being correct. Then, after 32s it notified CRNG init done and then goes into kernal panic from which I can see the above error message as last line. Screen output starts with panic stack trace:

panic from mount_block_root
mount_block_root from mount_root
mount_root from prepare_namespace
prepare_namespace from kernel_init_freeable
kernel_init_freeable from kernel_init
kernel_init from ret_from_fork

Apparently client is not able to read the root fs with this cmdline.txt:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=nfs root=/dev/nfs nfsroot=192.168.0.48:/nfs/client1 rw ip=dhcp roowait elevator=deadline

I've verified that the nfs share is actually accessible via nfs using a second raspi.

该提问来源于开源项目:raspberrypi/firmware

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

10条回答

  • weixin_39602967 weixin_39602967 4月前

    i have the same issue, after i installed Raspbian Stretch on the PXE-Server. adding ,vers=3 at the end of nfsroot= parameter in cmdline.txt fix that issue.

    
    wc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.0.1:/srv/nfs/rpi-12345678-root,vers=3 rw rootwait ip=dhcp elevator=deadline
    

    but it must be a bug or limitation of nfs mount ability of bootcode.bin or start.efi, because that PXE-Server is providing OS's for other x86 PC's as well, and those OS's are PXE booting and mounting nfsroot's without problems and without adding that additional ,vers=3 parameter. ubuntu-16.04.3, ubuntu-17.04, ubuntu-12-04.4, debian-live-9.1.0, raspberry pi jessie for x86 (2017-06-22), ...

    i saw, all those OS's are using at least vers=3 to mount nfsroot - even that old and crusty ubuntu-12-04.4. so bootcoode.bin, start.efi, or where ever the nfsroot mount happens, should use vers=3 as minimum as well.

    see here: https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=191224 and also here: https://github.com/raspberrypi/documentation/issues/719

    • Raspbian Jessie on the PXE-Server; RPi3 PXE network booting -> no problem;
    • Raspbian Jessie to Stretch update + upgrade on the PXE-Server; RPi3 PXE network booting -> no problem;
    • Raspbian Jessie to Stretch update + dist-upgrade on the PXE-Server; RPi3 not PXE network booting -> problem mounting nfsroot;
    • Raspbian Stretch on the PXE-Server; RPi3 not PXE network booting -> problem mounting nfsroot;

    PS.: my RPi3 is pxe network booting without any SD card. PPS.: my RPi-PXE-Server project

    PPPS.: for the pxe client, i tried out all the boot files from master, stable and next brunch of https://github.com/raspberrypi/firmware all shows up that same behavior.

    点赞 评论 复制链接分享
  • weixin_39605278 weixin_39605278 4月前

    Great, will test this asap.

    Raspbian Jessie to Stretch update + upgrade on the PXE-Server; RPi3 PXE network booting -> no problem; Raspbian Jessie to Stretch update + dist-upgrade on the PXE-Server; RPi3 not PXE network booting -> problem mounting nfsroot;

    That would mean that the client changes (i.e. bootcode and kernel) don't matter. I can see that PXE server on stretch is definitely NOT running NFSv2. So it might have been dropped by stretch?

    点赞 评论 复制链接分享
  • weixin_39602967 weixin_39602967 4月前

    yes, in my case, what the client is booting in (Raspbina Jessie or Stretch) makes no difference. what matters is, what is running on the Server

    with Raspbian Jessie as server, the client can use nfs version2, with Raspbian Stretch as server, the client must use nfs version 3 as minimum.

    maybe nfs version 2 on Raspbian Stretch was dropped for security reason.

    and somewhere it was forgotten to update the client nfs version from 2 up to 3 for the boot process. is it the kernel what is making the nfsroot mounting? then it is an issue of https://github.com/raspberrypi/linux ... isn't it ?

    点赞 评论 复制链接分享
  • weixin_39605278 weixin_39605278 4月前

    I can confirm that ,vers=3 takes care of the problem.

    is it the kernel what is making the nfsroot mounting?

    I think so. And the kernel supports v3 or the option wouldn't be working. So it may just be a topic of documenting to force v3 or make it the default boot option.

    点赞 评论 复制链接分享
  • weixin_39605278 weixin_39605278 4月前

    ping

    I'm leaving this open to get the tutorial updated at least.

    点赞 评论 复制链接分享
  • weixin_39617006 weixin_39617006 4月前

    Fixed in docs

    点赞 评论 复制链接分享
  • weixin_39889642 weixin_39889642 4月前

    Can we use version 4 NFS server instead in cmdline.txt? I'm trying to setup pi network boot server with docker in this gist, but version 3 NFS server in container behavior is weird, it won't work after container restart, and I can't figure out the reason. Others report version 4 works in docker. But after I replace vers=3 with vers=4 in /tftpboot/cmdline.txt got kernel panic, and the warning is same in this issues.

    点赞 评论 复制链接分享
  • weixin_39617006 weixin_39617006 4月前

    That's either because your kernel doesn't support v4 or your server...

    点赞 评论 复制链接分享
  • weixin_39889642 weixin_39889642 4月前

    It seems it's my nfs server container issues, after undo some changes, the service work properly now, and here is the docker-compose.yml : https://github.com/yangxuan8282/rpi3-pxe-docker . Both for x64 and armhf should works (the default tags of images is for armhf, and the amd64 tags is for x64).

    点赞 评论 复制链接分享
  • weixin_39605278 weixin_39605278 4月前

    Looking into the server with wireshark I may have an idea whats wrong:

    No.     Time           Source                Destination           Protocol Length Info
      20770 22.540054      192.168.0.66          192.168.0.48          NFS      82     V2 NULL Call (Reply In 20771)
    
    Frame 20770: 82 bytes on wire (656 bits), 82 bytes captured (656 bits)
    Ethernet II, Src: Raspberr_9d:c1:5a (b8:27:eb:9d:c1:5a), Dst: Raspberr_a9:e8:80 (b8:27:eb:a9:e8:80)
    Internet Protocol Version 4, Src: 192.168.0.66, Dst: 192.168.0.48
    User Datagram Protocol, Src Port: 867, Dst Port: 2049
    Remote Procedure Call, Type:Call XID:0x985adb98
        XID: 0x985adb98 (2556091288)
        Message Type: Call (0)
        RPC Version: 2
        Program: NFS (100003)
        Program Version: 2
        Procedure: NULL (0)
        [The reply to this request is in frame 20771]
        Credentials
        Verifier
    Network File System
        [Program Version: 2]
        [V2 Procedure: NULL (0)]
    
    No.     Time           Source                Destination           Protocol Length Info
      20771 22.540166      192.168.0.48          192.168.0.66          NFS      74     V2 NULL Reply (Call In 20770)
    
    Frame 20771: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Ethernet II, Src: Raspberr_a9:e8:80 (b8:27:eb:a9:e8:80), Dst: Raspberr_9d:c1:5a (b8:27:eb:9d:c1:5a)
    Internet Protocol Version 4, Src: 192.168.0.48, Dst: 192.168.0.66
    User Datagram Protocol, Src Port: 2049, Dst Port: 867
    Remote Procedure Call, Type:Reply XID:0x985adb98
        XID: 0x985adb98 (2556091288)
        Message Type: Reply (1)
        [Program: NFS (100003)]
        [Program Version: 2]
        [Procedure: NULL (0)]
        Reply State: accepted (0)
        [This is a reply to a request in frame 20770]
        [Time from request: 0.000112000 seconds]
        Verifier
        Accept State: remote can't support version # (2)
        Program Version (Minimum): 3
        Program Version (Maximum): 4
    

    It seems client is requesting NFS v2, server only supporting v3 and v4. Both systems are latest raspbian, so stretch.

    Using a second pi confirms this:

    pi:~ $ sudo mount -t nfs -o vers=2 keller:/nfs/client1 /mnt
    mount.nfs: Protocol not supported
    

    and so does the server itself:

    pi:~ $ rpcinfo -p localhost
       program vers proto   port  service
        100005    1   udp  35621  mountd
        100005    1   tcp  41791  mountd
        100005    2   udp  56907  mountd
        100005    2   tcp  48429  mountd
        100005    3   udp  39588  mountd
        100005    3   tcp  47781  mountd
        100003    3   tcp   2049  nfs
        100003    4   tcp   2049  nfs
        100227    3   tcp   2049
        100003    3   udp   2049  nfs
        100003    4   udp   2049  nfs
        100227    3   udp   2049
    ....
    

    Not a firmware issue?

    点赞 评论 复制链接分享

相关推荐