dpdk bind 失败

Question

问题遇到的现象和发生背景

需要使用DPDK的DSA功能，绑定vfio-pci失败

bind命令失败

./dpdk-devbind.py -b vfio-pci 0000:f6:01.0
Error: bind failed for 0000:f6:01.0 - Cannot bind to driver vfio-pci: [Errno 22] Invalid argument

# dmesg日志1
[  754.940840] vfio-pci 0000:f6:01.0: 8086:0b25 exists in vfio-pci device denylist, driver probing disallowed.
[  754.940847] vfio-pci: probe of 0000:f6:01.0 failed with error -22
[  754.940873] vfio-pci 0000:f6:01.0: 8086:0b25 exists in vfio-pci device denylist, driver probing disallowed.
[  754.940877] vfio-pci: probe of 0000:f6:01.0 failed with error -22

# dmesg日志2
dmesg | grep -e DMAR -e IOMMU
[    0.011416] ACPI: DMAR 0x0000000073E81000 0001E4 (v01 HPE    Server   00000001 1590 00000001)
[    0.011463] ACPI: Reserving DMAR table memory at [mem 0x73e81000-0x73e811e3]
[    0.318307] DMAR: IOMMU enabled
[    0.318309] DMAR: Enable scalable mode if hardware supports
[    0.318410] DMAR: IOMMU enabled
[    0.318411] DMAR: Enable scalable mode if hardware supports
[    0.818240] DMAR: Host address width 46
[    0.818242] DMAR: DRHD base: 0x000000b0ffc000 flags: 0x0
[    0.818247] DMAR: dmar0: reg_base_addr b0ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[    0.818251] DMAR: DRHD base: 0x000000c37fc000 flags: 0x0
[    0.818254] DMAR: dmar1: reg_base_addr c37fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[    0.818258] DMAR: DRHD base: 0x000000d5ffc000 flags: 0x0
[    0.818261] DMAR: dmar2: reg_base_addr d5ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[    0.818264] DMAR: DRHD base: 0x000000e87fc000 flags: 0x0
[    0.818268] DMAR: dmar3: reg_base_addr e87fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[    0.818271] DMAR: DRHD base: 0x000000faffc000 flags: 0x0
[    0.818274] DMAR: dmar4: reg_base_addr faffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[    0.818277] DMAR: DRHD base: 0x000000fb7fc000 flags: 0x0
[    0.818283] DMAR: dmar5: reg_base_addr fb7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
[    0.818286] DMAR: DRHD base: 0x0000009e3fc000 flags: 0x1
[    0.818289] DMAR: dmar6: reg_base_addr 9e3fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[    0.818293] DMAR: RMRR base: 0x00000076668000 end: 0x0000007666afff
[    0.818295] DMAR: ATSR flags: 0x0
[    0.818297] DMAR: RHSA base: 0x0000009e3fc000 proximity domain: 0x0
[    0.818299] DMAR: RHSA base: 0x000000b0ffc000 proximity domain: 0x0
[    0.818301] DMAR: RHSA base: 0x000000c37fc000 proximity domain: 0x0
[    0.818303] DMAR: RHSA base: 0x000000d5ffc000 proximity domain: 0x0
[    0.818304] DMAR: RHSA base: 0x000000e87fc000 proximity domain: 0x0
[    0.818306] DMAR: RHSA base: 0x000000faffc000 proximity domain: 0x0
[    0.818307] DMAR: RHSA base: 0x000000fb7fc000 proximity domain: 0x0
[    0.818309] DMAR: SATC flags: 0x0
[    0.818311] DMAR-IR: IOAPIC id 8 under DRHD base  0x9e3fc000 IOMMU 6
[    0.818314] DMAR-IR: HPET id 0 under DRHD base 0x9e3fc000
[    0.818316] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.819508] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.742891] DMAR: IOMMU feature pasid inconsistent
[    1.742893] DMAR: IOMMU feature prs inconsistent
[    1.742895] DMAR: IOMMU feature pasid inconsistent
[    1.742896] DMAR: IOMMU feature prs inconsistent
[    1.742898] DMAR: IOMMU feature pasid inconsistent
[    1.742899] DMAR: IOMMU feature prs inconsistent
[    1.742900] DMAR: IOMMU feature pasid inconsistent
[    1.742901] DMAR: IOMMU feature prs inconsistent
[    1.742903] DMAR: IOMMU feature pasid inconsistent
[    1.742904] DMAR: IOMMU feature prs inconsistent
[    1.742905] DMAR: IOMMU feature pasid inconsistent
[    1.742906] DMAR: IOMMU feature prs inconsistent
[    1.742908] DMAR: dmar5: Using Queued invalidation
[    1.742917] DMAR: dmar4: Using Queued invalidation
[    1.742919] DMAR: dmar3: Using Queued invalidation
[    1.742921] DMAR: dmar2: Using Queued invalidation
[    1.742923] DMAR: dmar1: Using Queued invalidation
[    1.742929] DMAR: dmar0: Using Queued invalidation
[    1.742931] DMAR: dmar6: Using Queued invalidation
[    1.795468] DMAR: Intel(R) Virtualization Technology for Directed I/O

操作环境、软件版本等信息

uname -a

Linux spr153 5.15.163 #2 SMP Fri Sep 27 11:09:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

./dpdk-devbind.py -s

Other DMA devices
=================
0000:f6:01.0 'Device 0b25' unused=idxd,vfio-pci

# lspci 打印设备信息
f6:01.0 System peripheral: Intel Corporation Device 0b25
        Subsystem: Intel Corporation Device 0000
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 143
        Region 0: Memory at 21bffff20000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at 21bffff00000 (64-bit, prefetchable) [size=128K]
        Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0
                        ExtTag+ RBE+ FLReset+
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+ LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
        Capabilities: [80] MSI-X: Enable- Count=9 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [90] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [160 v1] Transaction Processing Hints
                Device specific mode supported
                Steering table in TPH capability structure
        Capabilities: [170 v1] Virtual Channel
                Caps:   LPEVC=1 RefClk=100ns PATEntryBits=1
                Arb:    Fixed+ WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=fd
                        Status: NegoPending- InProgress-
                VC1:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=1 ArbSelect=Fixed TC/VC=02
                        Status: NegoPending- InProgress-
        Capabilities: [200 v1] Designated Vendor-Specific: Vendor=8086 ID=0005 Rev=0 Len=24 <?>
        Capabilities: [220 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [230 v1] Process Address Space ID (PASID)
                PASIDCap: Exec- Priv+, Max PASID Width: 14
                PASIDCtl: Enable+ Exec- Priv+
        Capabilities: [240 v1] Page Request Interface (PRI)
                PRICtl: Enable+ Reset-
                PRISta: RF- UPRGI- Stopped+
                Page Request Capacity: 00000200, Page Request Allocation: 00000200
        Kernel modules: idxd

我想要达到的结果

./dpdk-devbind.py -b vfio-pci 0000:f6:01.0 成功执行

Answer 1

🕒下午好，题主！目前：🟢[在线]
📢参考通义千问、由郭老师整理提供！
📣今日诗号：彼采葛兮，一日不见，如三月兮！

您遇到的问题是DPDK绑定VFIO-PCI驱动失败，原因是设备0000:f6:01.0被列入了VFIO-PCI的黑名单。根据您提供的信息，我总结出以下解决步骤和代码示例：

1. 确认问题根源:

• vfio-pci denylist: dmesg日志显示设备被列入vfio-pci denylist。这通常是因为其他驱动程序正在使用该设备，或者系统配置禁止vfio-pci接管该设备。
• 设备驱动: 从lspci信息可以看到，设备0000:f6:01.0当前的驱动是idxd。您需要禁用idxd驱动，才能将VFIO-PCI绑定到该设备。

2. 解决方案:

以下是解决此问题的步骤：

a. 查找并禁用 idxd 驱动:

• 确认 idxd 驱动是否已加载:

  lsmod | grep idxd
  

• 如果已加载，卸载 idxd 驱动:

  sudo modprobe -r idxd
  

• 阻止 idxd 驱动自动加载 (需要root权限):
  • 找到 idxd 模块的配置文件:

  find /lib/modules/$(uname -r) -type f -name idxd.ko
  

  • 创建一个配置文件来阻止加载:

  sudo bash -c "echo 'blacklist idxd' > /etc/modprobe.d/blacklist-idxd.conf"
  

b. 将设备从PCI设备 denylist 中移除:

• vfio-pci 的 denylist 通常在 /etc/modprobe.d/vfio.conf 或 /etc/modprobe.d/vfio-pci.conf 中配置。
• 打开配置文件并找到包含设备 PCI ID (8086:0b25) 的行，将其注释掉或者删除。
• 示例:

  # options vfio-pci ids=10de:10f0,8086:0b25
  

  将其改为:

  options vfio-pci ids=10de:10f0
  

c. 重新加载 VFIO-PCI 模块:

sudo modprobe -r vfio-pci
sudo modprobe vfio-pci

d. 绑定 VFIO-PCI 驱动:

./dpdk-devbind.py -b vfio-pci 0000:f6:01.0

3. 验证:

• 使用 lspci -v 命令查看设备 0000:f6:01.0 的驱动程序是否已成功绑定为 vfio-pci。

注意:

• 以上步骤需要 root 权限。
• 请确保在执行上述步骤之前备份您的系统配置文件。
• 如果问题仍然存在，请提供完整的 dmesg 日志和 lspci -vvv 命令的输出，以便我更好地帮助您解决问题.

希望以上信息对您有所帮助！