Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot add MAC address to a VF interface #1125

Open
lucascgo opened this issue Apr 24, 2024 · 1 comment
Open

Cannot add MAC address to a VF interface #1125

lucascgo opened this issue Apr 24, 2024 · 1 comment

Comments

@lucascgo
Copy link

Hi, I have encountered an issue in TRex v3.04 with the Broadcom NetXtreme-E NICs.

If I bind the PFs to DPDK I can run the TRex, but if I bind the VFs I get this error:

bnxt_stats_reset_op(): Device Initialization not complete!
bnxt_mac_addr_add_op(): Cannot add MAC address to a VF interface
Failed setting MAC for port 0
Killing Scapy server... Scapy server is killed

Using TestPMD I can reach the VFs.

The full log with TRex is bellow (the error is at the end):

$ sudo ./t-rex-64 -i -v 7 --cfg /etc/trex_cfg.yaml
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
Starting  TRex v3.04 please wait  ...
Using configuration file /etc/trex_cfg.yaml
 port limit     :  not configured
 port_bandwidth_gb    :  10
 port_speed           :  0
 port_mtu             :  0
 if_mask        : None
 is low-end : 0
 stack type :
 thread_per_dual_if      : 1
 if        :  65:02.0, 65:03.0,
 enable_zmq_pub :  1
 zmq_pub_port   :  4500
 m_zmq_rpc_port    :  4501
 src     : 00:00:00:00:00:00
 dest    : 00:00:00:00:00:00
 src     : 00:00:00:00:00:00
 dest    : 00:00:00:00:00:00
 memory per 2x10G ports
 MBUF_64                                   : 16380
 MBUF_128                                  : 8190
 MBUF_256                                  : 8190
 MBUF_512                                  : 8190
 MBUF_1024                                 : 8190
 MBUF_2048                                 : 4095
 MBUF_4096                                 : 128
 MBUF_9K                                   : 512
 TRAFFIC_MBUF_64                           : 65520
 TRAFFIC_MBUF_128                          : 32760
 TRAFFIC_MBUF_256                          : 8190
 TRAFFIC_MBUF_512                          : 8190
 TRAFFIC_MBUF_1024                         : 8190
 TRAFFIC_MBUF_2048                         : 32760
 TRAFFIC_MBUF_4096                         : 128
 TRAFFIC_MBUF_9K                           : 512
 MBUF_DP_FLOWS                             : 524288
 MBUF_GLOBAL_FLOWS                         : 5120
 master   thread  : 0
 rx  thread  : 15
 dual_if : 0
    socket  : 0
   [   1   2   3   4   5   6   7   8   9   10   11   12   13   14     ]
CTimerWheelYamlInfo does not exist
 flags           : 8010f00
 write_file      : 0
 verbose         : 7
 realtime        : 1
 flip            : 0
 cores           : 1
 single core     : 0
 flow-flip       : 0
 no clean close  : 0
 zmq_publish     : 1
 vlan mode       : 0
 client_cfg      : 0
 mbuf_cache_disable  : 0
 cfg file        :
 mac file        :
 out file        :
 client cfg file :
 duration        : 0
 factor          : 1
 mbuf_factor     : 1
 latency         : 0 pkt/sec
 zmq_port        : 4500
 telnet_port     : 4501
 expected_ports  : 2
 tw_bucket_usec  : 20.000000 usec
 tw_buckets      : 1024 usec
 tw_levels       : 3 usec
 port : 0 dst:00:00:00:00:00:00  src:00:00:00:00:00:00
 port : 1 dst:00:00:00:00:00:00  src:00:00:00:00:00:00
 port : 2 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 3 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 4 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 5 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 6 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 7 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 8 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 9 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 10 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 11 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 12 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 13 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 14 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 15 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 16 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 17 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 18 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 19 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 20 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 21 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 22 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 23 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 24 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 25 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 26 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 27 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 28 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 29 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 30 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 31 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 Total Memory :
 MBUF_64                                   : 81900
 MBUF_128                                  : 40950
 MBUF_256                                  : 16380
 MBUF_512                                  : 16380
 MBUF_1024                                 : 16380
 MBUF_2048                                 : 36855
 MBUF_4096                                 : 1024
 MBUF_DP_FLOWS                             : 524288
 MBUF_GLOBAL_FLOWS                         : 5120
 get_each_core_dp_flows                    : 524288
 Total memory                              :     248.40 Mbytes
 core_list : 0,15,1
 sockets : 0
 active sockets : 1
 ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 phy   |   virt
 1      1
DPDK args
 xx  -l  0,15,1  -n  4  --log-level  8  --main-lcore  0  -a  0000:65:02.0  -a  0000:65:03.0  --legacy-mem
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:02.0 (socket 0)
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:03.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
                 input : [65:02.0, 65:03.0]
                 dpdk : [0000:65:02.0, 0000:65:03.0]
             pci_scan : [0000:65:02.0, 0000:65:03.0]
                  map : [ 0, 1]
 TRex port mapping
 -----------------
 TRex vport: 0 dpdk_rte_eth: 0
 TRex vport: 1 dpdk_rte_eth: 1
 set driver name net_bnxt
 driver capability  : TCP_UDP_OFFLOAD  TSO  LRO
 set dpdk queues mode to ONE_QUE
 DPDK devices 2 : 2
-----
 0 : vdev 0000:65:02.0
 1 : vdev 0000:65:03.0
-----
 Number of ports found: 2


if_index : 0
driver name : net_bnxt
min_rx_bufsize : 1
max_rx_pktlen  : 9600
max_rx_queues  : 63
max_tx_queues  : 63
max_mac_addrs  : 128
rx_offload_capa : 0xd265f
tx_offload_capa : 0x19faf
rss reta_size   : 128
flow_type_rss   : 0xc000000000d34
tx_desc_max     : 4096
tx_desc_min     : 16
rx_desc_max     : 8192
rx_desc_min     : 16
zmq publisher at: tcp://*:4500
 rx_data_q_num : 1
 rx_drop_q_num : 0
 rx_dp_q_num   : 0
 rx_que_total : 1
 --
 rx_desc_num_data_q   : 4096
 rx_desc_num_drop_q   : 4096
 rx_desc_num_dp_q     : 0
 total_desc           : 4096
 --
 tx_desc_num     : 1024
port 0 desc: NetXtreme-E Ethernet Virtual Function
 rx_qid: 0 (4096)
bnxt_stats_reset_op(): Device Initialization not complete!
bnxt_mac_addr_add_op(): Cannot add MAC address to a VF interface
Failed setting MAC for port 0
Killing Scapy server... Scapy server is killed

Below some other infos:

 Open Source Components / Libraries 
 DPDK       (BSD)       
 YAML-CPP   (BSD)       
 JSONCPP    (MIT)       
 BPF        (BSD)       
 HDR-HISTOGRAM-C (CC0)  
 
 Open Source Binaries 
 ZMQ        (LGPL v3plus) 
 
 Version : v3.04   
 DPDK version : DPDK 23.03.0   
 User    : root   
 Date    : Apr 24 2024 , 20:39:54 
 Uuid    : 393570d6-027a-11ef-900d-73f27c253397    
 Git SHA : 64cd404ed8614ad9d5e5a86f0334100e134647d8    

Compiled with GCC     :   9.4.0
Compiled with glibc   :   2.31 (host: 2.31)
Sanitized image       :   no

$ sudo ./dpdk_setup_ports.py -t
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| ID | NUMA |   PCI   |        MAC        |                             Name                             |  Driver  | Linux IF  |  Active  |
+====+======+=========+===================+==============================================================+==========+===========+==========+
| 0  | 0    | 04:00.0 | 2c:ea:7f:b6:8a:1e | NetXtreme BCM5720 2-port Gigabit Ethernet PCIe               | tg3      | eno1      | *Active* |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 1  | 0    | 04:00.1 | 2c:ea:7f:b6:8a:1f | NetXtreme BCM5720 2-port Gigabit Ethernet PCIe               | tg3      | eno2      |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 2  | 0    | 65:00.0 | b0:26:28:79:e4:20 | BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller | bnxt_en  | ens2f0np0 |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 3  | 0    | 65:00.1 | b0:26:28:79:e4:21 | BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller | bnxt_en  | ens2f1np1 |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 4  | 0    | 65:02.0 | 62:52:b3:8c:eb:0f | NetXtreme-E Ethernet Virtual Function                        | vfio-pci |           |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 5  | 0    | 65:03.0 | 8a:82:0e:c2:bf:9e | NetXtreme-E Ethernet Virtual Function                        | vfio-pci |           |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+

$ sudo ./dpdk_setup_ports.py -s

Network devices using DPDK-compatible driver
============================================
0000:65:02.0 'NetXtreme-E Ethernet Virtual Function' drv=vfio-pci unused=bnxt_en,igb_uio,uio_pci_generic
0000:65:03.0 'NetXtreme-E Ethernet Virtual Function' drv=vfio-pci unused=bnxt_en,igb_uio,uio_pci_generic

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic 
0000:65:00.0 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller' if=ens2f0np0 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic 
0000:65:00.1 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller' if=ens2f1np1 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

/etc/trex_cfg.yaml

### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['65:02.0', '65:03.0']
  port_info:
      - ip: 10.60.1.45
        default_gw: 10.60.1.1
      - ip: 10.30.1.45
        default_gw: 10.30.1.1

  platform:
      master_thread_id: 0
      latency_thread_id: 15
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14]
@lucascgo
Copy link
Author

lucascgo commented May 9, 2024

I tried other versions of TRex and the last version without error in the bxnt driver with the VFs is v2.87.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
1 participant
@lucascgo