Build For Azure Ubuntu (mlx5)

TRex package is built with DPDK mlx5/tap driver bind to CentOs kernel headers. It won’t work with a different kernel (e.g. Ubuntu that has newer kernel) To support different kernel API version the following steps should be followed

Build Ubuntu VM in Azure

az vm create --resource-group mygroup --location eastus --name TrexUbuntuOs --image UbuntuLTS --size Standard_F16s_v2 --admin-username testing --admin-password example --nics myNIC1 myNIC2 myNIC3

Follow Azure DPDK setup steps

see (https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk)

sudo apt-get update
sudo apt-get install -y librdmacm-dev librdmacm1 build-essential libnuma-dev libmnl-dev

setup huge pages on reboot

sudo vi /etc/default/grub
# GRUB_CMDLINE_LINUX="hugepages=4096"
sudo update-grub
sudo vi /etc/fstab
# nodev /mnt/huge hugetlbfs defaults 0 0
# prevent renaming of dtapX

sudo vi /etc/systemd/network/10-dtap.link
# [Match]
# OriginalName=dtap*
# [Link]
# NamePolicy=kernel
# load drivers on reboot

sudo vi /etc/modules-load.d/modules.conf
# ib_uverbs
# mlx4_ib
# mlx5_ib
sudo reboot

After reboot

Validate huge pages and drivers loaded

cat /proc/meminfo | grep Huge
lsmod | grep ib_uverbs

Follow Azure DPDK build

Build steps to get autoconf files

tar xJf dpdk-19.05.tar.xz
cd ~/dpdk-19.05
make config T=x86_64-native-linuxapp-gcc
sed -ri 's,(MLX._PMD=)n,\1y,' build/.config
make

Validate DPDK with testpmd

cd ~/dpdk-19.05/build/app
sudo ./testpmd -w 0002:00:02.0 -w 0003:00:02.0 --vdev="net_vdev_netvsc0,iface=eth1" --vdev="net_vdev_netvsc1,iface=eth2" -- -i
# validate dtapX exist (no renaming)
ifconfig -a | grep flags
dtap0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
dtap1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
eth1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
eth2: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
rename5: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
rename6: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500

Copy autoconf files

From DPDK to use with TREX build

cd ~/dpdk-19.05/
find . -name "*autoconf.h"
./build/build/drivers/net/tap/tap_autoconf.h
./build/build/drivers/net/mlx4/mlx4_autoconf.h
./build/build/drivers/net/mlx5/mlx5_autoconf.h
cp ./build/build/drivers/net/tap/tap_autoconf.h ~/.
cp ./build/build/drivers/net/mlx4/mlx4_autoconf.h ~/.
cp ./build/build/drivers/net/mlx5/mlx5_autoconf.h ~/.

Setup for build with TREX

cd ~
sudo apt-get install -y python3-distutils
sudo apt install zlib1g-dev
git clone https://github.com/cisco-system-traffic-generator/trex-core.git # better to take a latest label and not the latest
# To compile DPDK (TAP, MLX4, MLX5) on the machine natively
# Copy in the autoconf files from DPDK build.
# In TAP dir (src/dpdk/drivers/net/tap) switch all #include <linux_tap/> to <linux/>
# Manually edit linux_dpdk/ws_main.py to set check_ofed to true and remove MLX4 define
# (see diffs below)

Copy the files

cd ~/trex-core
cp ~/tap_autoconf.h src/dpdk/drivers/net/tap/tap_autoconf.h
cp ~/mlx4_autoconf.h src/dpdk/drivers/net/mlx4/mlx4_autoconf.h
cp ~/mlx5_autoconf.h src/dpdk/drivers/net/mlx5/mlx5_autoconf.h
cd src/dpdk/drivers/net/tap
sed -i 's/linux_tap/linux/g' *
cd ~/trex-core

status

git status

modified:   linux_dpdk/ws_main.py
modified:   src/dpdk/drivers/net/mlx4/mlx4_autoconf.h
modified:   src/dpdk/drivers/net/mlx5/mlx5_autoconf.h
modified:   src/dpdk/drivers/net/tap/rte_eth_tap.c
modified:   src/dpdk/drivers/net/tap/rte_eth_tap.h
modified:   src/dpdk/drivers/net/tap/tap_autoconf.h
modified:   src/dpdk/drivers/net/tap/tap_netlink.h
modified:   src/dpdk/drivers/net/tap/tap_tcmsgs.h

Diff

  diff --git a/linux_dpdk/ws_main.py b/linux_dpdk/ws_main.py
  index 7d685f61..82656fc4 100755
  --- a/linux_dpdk/ws_main.py
  +++ b/linux_dpdk/ws_main.py
  @@ -209,6 +209,7 @@ def check_ofed(ctx):
       ofed_ver= 42
       ofed_ver_show= '4.2'
  +    return True
       if not os.path.isfile(ofed_info):
           ctx.end_msg('not found', 'YELLOW')
           return False
  @@ -1478,8 +1479,6 @@ class build_option:
              flags += ['-DNDEBUG'];
          else:
              flags += ['-UNDEBUG'];
  -        if bld.env.OFED_OK:
  -            flags += ['-DHAVE_IBV_MLX4_WQE_LSO_SEG=1']
           return (flags)
       def get_common_flags (self):

Build Trex after modifications for native build on Ubuntu

cd linux_dpdk
./b configure
./b build

Create trex_cfg.yaml for the system

Typical setup routes

sudo route add -net 16.0.0.0 netmask 255.0.0.0 gw 10.90.130.202
sudo route add -net 48.0.0.0 netmask 255.0.0.0 gw 10.90.23.202

Turn off TSO etc….

sudo ethtool -K eth1 tso off gro off gso off
sudo ethtool -K eth2 tso off gro off gso off
sudo ethtool -K rename5 tso off gro off gso off
sudo ethtool -K rename6 tso off gro off gso off

Run TREX

cd ~/trex-core/scripts
sudo ./t-rex-64 -i -c 1 -v 7 --no-ofed-check

Run tui/bench

stty cols 111 rows 45
cd ~/trex-core/scripts
./trex-console
trex> tui
tui> start -f stl/bench.py -t vm=cached,size=1518 -m 230kpps --port 0 1 --force
-----

.Run ndr script
 ----
./ndr.bash --python3 --port 0 1 -c 1 1 -fe cached  -x 10 -t 60 -v --force-map -ld --pdr 0.01 -size 1514 -ti 'PDR_0.01_1518_byte_bi-directional' --bi-dir