问题遇到的现象和发生背景
需要使用DPDK的DSA功能，绑定vfio-pci失败
bind命令失败
./dpdk-devbind.py -b vfio-pci 0000:f6:01.0
Error: bind failed for 0000:f6:01.0 - Cannot bind to driver vfio-pci: [Errno 22] Invalid argument
# dmesg日志1
[ 754.940840] vfio-pci 0000:f6:01.0: 8086:0b25 exists in vfio-pci device denylist, driver probing disallowed.
[ 754.940847] vfio-pci: probe of 0000:f6:01.0 failed with error -22
[ 754.940873] vfio-pci 0000:f6:01.0: 8086:0b25 exists in vfio-pci device denylist, driver probing disallowed.
[ 754.940877] vfio-pci: probe of 0000:f6:01.0 failed with error -22
# dmesg日志2
dmesg | grep -e DMAR -e IOMMU
[ 0.011416] ACPI: DMAR 0x0000000073E81000 0001E4 (v01 HPE Server 00000001 1590 00000001)
[ 0.011463] ACPI: Reserving DMAR table memory at [mem 0x73e81000-0x73e811e3]
[ 0.318307] DMAR: IOMMU enabled
[ 0.318309] DMAR: Enable scalable mode if hardware supports
[ 0.318410] DMAR: IOMMU enabled
[ 0.318411] DMAR: Enable scalable mode if hardware supports
[ 0.818240] DMAR: Host address width 46
[ 0.818242] DMAR: DRHD base: 0x000000b0ffc000 flags: 0x0
[ 0.818247] DMAR: dmar0: reg_base_addr b0ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 0.818251] DMAR: DRHD base: 0x000000c37fc000 flags: 0x0
[ 0.818254] DMAR: dmar1: reg_base_addr c37fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 0.818258] DMAR: DRHD base: 0x000000d5ffc000 flags: 0x0
[ 0.818261] DMAR: dmar2: reg_base_addr d5ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 0.818264] DMAR: DRHD base: 0x000000e87fc000 flags: 0x0
[ 0.818268] DMAR: dmar3: reg_base_addr e87fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 0.818271] DMAR: DRHD base: 0x000000faffc000 flags: 0x0
[ 0.818274] DMAR: dmar4: reg_base_addr faffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 0.818277] DMAR: DRHD base: 0x000000fb7fc000 flags: 0x0
[ 0.818283] DMAR: dmar5: reg_base_addr fb7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
[ 0.818286] DMAR: DRHD base: 0x0000009e3fc000 flags: 0x1
[ 0.818289] DMAR: dmar6: reg_base_addr 9e3fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
[ 0.818293] DMAR: RMRR base: 0x00000076668000 end: 0x0000007666afff
[ 0.818295] DMAR: ATSR flags: 0x0
[ 0.818297] DMAR: RHSA base: 0x0000009e3fc000 proximity domain: 0x0
[ 0.818299] DMAR: RHSA base: 0x000000b0ffc000 proximity domain: 0x0
[ 0.818301] DMAR: RHSA base: 0x000000c37fc000 proximity domain: 0x0
[ 0.818303] DMAR: RHSA base: 0x000000d5ffc000 proximity domain: 0x0
[ 0.818304] DMAR: RHSA base: 0x000000e87fc000 proximity domain: 0x0
[ 0.818306] DMAR: RHSA base: 0x000000faffc000 proximity domain: 0x0
[ 0.818307] DMAR: RHSA base: 0x000000fb7fc000 proximity domain: 0x0
[ 0.818309] DMAR: SATC flags: 0x0
[ 0.818311] DMAR-IR: IOAPIC id 8 under DRHD base 0x9e3fc000 IOMMU 6
[ 0.818314] DMAR-IR: HPET id 0 under DRHD base 0x9e3fc000
[ 0.818316] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.819508] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.742891] DMAR: IOMMU feature pasid inconsistent
[ 1.742893] DMAR: IOMMU feature prs inconsistent
[ 1.742895] DMAR: IOMMU feature pasid inconsistent
[ 1.742896] DMAR: IOMMU feature prs inconsistent
[ 1.742898] DMAR: IOMMU feature pasid inconsistent
[ 1.742899] DMAR: IOMMU feature prs inconsistent
[ 1.742900] DMAR: IOMMU feature pasid inconsistent
[ 1.742901] DMAR: IOMMU feature prs inconsistent
[ 1.742903] DMAR: IOMMU feature pasid inconsistent
[ 1.742904] DMAR: IOMMU feature prs inconsistent
[ 1.742905] DMAR: IOMMU feature pasid inconsistent
[ 1.742906] DMAR: IOMMU feature prs inconsistent
[ 1.742908] DMAR: dmar5: Using Queued invalidation
[ 1.742917] DMAR: dmar4: Using Queued invalidation
[ 1.742919] DMAR: dmar3: Using Queued invalidation
[ 1.742921] DMAR: dmar2: Using Queued invalidation
[ 1.742923] DMAR: dmar1: Using Queued invalidation
[ 1.742929] DMAR: dmar0: Using Queued invalidation
[ 1.742931] DMAR: dmar6: Using Queued invalidation
[ 1.795468] DMAR: Intel(R) Virtualization Technology for Directed I/O
操作环境、软件版本等信息
uname -a
Linux spr153 5.15.163 #2 SMP Fri Sep 27 11:09:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
./dpdk-devbind.py -s
Other DMA devices
=================
0000:f6:01.0 'Device 0b25' unused=idxd,vfio-pci
# lspci 打印设备信息
f6:01.0 System peripheral: Intel Corporation Device 0b25
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
NUMA node: 0
IOMMU group: 143
Region 0: Memory at 21bffff20000 (64-bit, prefetchable) [size=64K]
Region 2: Memory at 21bffff00000 (64-bit, prefetchable) [size=128K]
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+ FLReset+
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+ LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
Capabilities: [80] MSI-X: Enable- Count=9 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [90] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [150 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [160 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Capabilities: [170 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=fd
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=1 ArbSelect=Fixed TC/VC=02
Status: NegoPending- InProgress-
Capabilities: [200 v1] Designated Vendor-Specific: Vendor=8086 ID=0005 Rev=0 Len=24 <?>
Capabilities: [220 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [230 v1] Process Address Space ID (PASID)
PASIDCap: Exec- Priv+, Max PASID Width: 14
PASIDCtl: Enable+ Exec- Priv+
Capabilities: [240 v1] Page Request Interface (PRI)
PRICtl: Enable+ Reset-
PRISta: RF- UPRGI- Stopped+
Page Request Capacity: 00000200, Page Request Allocation: 00000200
Kernel modules: idxd
我想要达到的结果
./dpdk-devbind.py -b vfio-pci 0000:f6:01.0 成功执行
