Rare project developers

[mc36 <>]

hi,
heard during the today vc that you have plans to build a dpdk flavor... 
sounds crazy good!
i'm writing to you to show you an example on how to start it...
in the below examples, i'll use the http://dl.nop.hu/rtr.vmdk image plus the 
contents of
http://src.freertr.org/binImg/rtr.extra extracted to the root of that disk 
image...
after doing so, you can start it and do the bottom of freertr.org, but 
instead of
p4emu, you can specify p4dpdk or p4xpd...i personally would suggest to you if 
you decide
to bake a software router then keep the possibility to run these too: the 
p4emu uses
only libpcap so it'll run anywhere... the p4xdp is an ebpf/xdp forwarder, the 
exact
opposite what p4dpdk does... so here is how it ended after specifying the 
dpdk flavor:

yourname#flash type hwdet-all.sh
#!/bin/sh

cd /rtr/
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6                # these 
generate packets alone which fools the dataplanes
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6
ip link set lo up mtu 65535
ip addr add 127.0.0.1/8 dev lo
ip addr add ::1/128 dev lo
ulimit -c unlimited                                              # needed to 
have core dumps
#modprobe -r kvm_intel
#modprobe kvm_intel nested=1
#echo 1 > /sys/kernel/mm/ksm/run

### macs ###
echo starting macs.
# ens3 52:54:00:12:34:56 #

### interfaces ###                                     ##### just some 
interface initialization to be sure
echo starting interfaces.
ip link set ens3 up multicast on promisc on mtu 1500
ethtool -K ens3 rx off
ethtool -K ens3 tx off
ethtool -K ens3 sg off
ethtool -K ens3 tso off
ethtool -K ens3 ufo off
ethtool -K ens3 gso off
ethtool -K ens3 gro off
ethtool -K ens3 lro off
ethtool -K ens3 rxvlan off
ethtool -K ens3 txvlan off
ethtool -K ens3 ntuple off
ethtool -K ens3 rxhash off
ethtool --set-eee ens3 eee off


### lines ###
echo starting lines.
start-stop-daemon -S -b -x /rtr/hwdet-lin1.sh


### dataplane ###
ip link add veth1a type veth peer name veth0                              # 
the ethernet2 provided by tapint in tofino with 10.255.255.1/24
ip link set veth0 name veth1b
ip link set veth1a up multicast on promisc on mtu 1500
ethtool -K veth1a rx off
ethtool -K veth1a tx off
ethtool -K veth1a sg off
ethtool -K veth1a tso off
ethtool -K veth1a ufo off
ethtool -K veth1a gso off
ethtool -K veth1a gro off
ethtool -K veth1a lro off
ethtool -K veth1a rxvlan off
ethtool -K veth1a txvlan off
ethtool -K veth1a ntuple off
ethtool -K veth1a rxhash off
ethtool --set-eee veth1a eee off
ip link set veth1b up multicast on promisc on mtu 8192
ethtool -K veth1b rx off
ethtool -K veth1b tx off
ethtool -K veth1b sg off
ethtool -K veth1b tso off
ethtool -K veth1b ufo off
ethtool -K veth1b gso off
ethtool -K veth1b gro off
ethtool -K veth1b lro off
ethtool -K veth1b rxvlan off
ethtool -K veth1b txvlan off
ethtool -K veth1b ntuple off
ethtool -K veth1b rxhash off
ethtool --set-eee veth1b eee off
ip link set veth1a up address 00:00:11:11:22:22 mtu 1500
ip addr add 10.255.255.1/24 dev veth1a
ip route add 0.0.0.0/0 via 10.255.255.254 dev veth1a
echo 0 > /proc/sys/net/ipv6/conf/veth1a/disable_ipv6
#modprobe uio_pci_generic
echo 128 > /proc/sys/vm/nr_hugepages                                      # 
dpdk needs hugepages, the amount depends on the # of interfaces
modprobe vfio-pci                                                         # a 
pci passthrough alike method to steal the interfaces from kernel
dpdk-devbind.py -b vfio-pci ens3                                          # 
bind the interfaces to this vfio driver to used by dpdk pmds
ip link add veth0a type veth peer name veth0                              # 
the cpuport equivalent but here over a veth pair
ip link set veth0 name veth0b
ip link set veth0a up multicast on promisc on mtu 8192
ethtool -K veth0a rx off
ethtool -K veth0a tx off
ethtool -K veth0a sg off
ethtool -K veth0a tso off
ethtool -K veth0a ufo off
ethtool -K veth0a gso off
ethtool -K veth0a gro off
ethtool -K veth0a lro off
ethtool -K veth0a rxvlan off
ethtool -K veth0a txvlan off
ethtool -K veth0a ntuple off
ethtool -K veth0a rxhash off
ethtool --set-eee veth0a eee off
ip link set veth0b up multicast on promisc on mtu 8192
ethtool -K veth0b rx off
ethtool -K veth0b tx off
ethtool -K veth0b sg off
ethtool -K veth0b tso off
ethtool -K veth0b ufo off
ethtool -K veth0b gso off
ethtool -K veth0b gro off
ethtool -K veth0b lro off
ethtool -K veth0b rxvlan off
ethtool -K veth0b txvlan off
ethtool -K veth0b ntuple off
ethtool -K veth0b rxhash off
ethtool --set-eee veth0b eee off

### main ###
echo starting main.
start-stop-daemon -S -b -x /rtr/hwdet-main.sh
exit 0

yourname#
yourname#flash type rtr-hw.txt
hwid xxx
tcp2vrf 23 host 23
line tty1 127.0.0.1 20022 127.0.0.1 20021
tcp2vrf 2323 p4 23 127.0.0.1
tcp2vrf 9080 p4 9080 127.0.0.1
proc p4emu /rtr/p4dpdk.bin --vdev=net_af_packet0,iface=ens3 
--vdev=net_af_packet1,iface=veth1b --vdev=net_af_packet2,iface=veth0b -- 
127.0.0.1 9080 2
int eth0 eth - 127.0.0.1 19999 127.0.0.1 19998
proc cpuport /rtr/pcapInt.bin veth0a 19998 127.0.0.1 19999 127.0.0.1

yourname#

the trick here is that you start p4dpdk and list the interfaces with the 
dpdk's eal --vdev arguments...
after the --, the p4dpdk.c's arguments follow... the freerouter control 
channel, and the index of cpuport...
here it's 2 because veth0b got interface index 2 (numbering starts from 0 as 
you can spot..)
so the bare minimum is to 'detect' the interfaces, and list them in the 
rtr-hw.txt...
similar parameters needed to bring up the p4xdp and p4emu dataplanes, except 
that you dont have to bind to vfio...

regardind the dpdk dataplane, one can continue the arguments by specifying 
which cpu core receives or transmits for the given nic...
i've already written about these extra parameters here about a year ago:
https://groups.io/g/freertr/topic/rare_users_freertr_about/93231625?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,100,93231625,previd%3D1661412819560986972,nextid%3D1661089187920291335&previd=1661412819560986972&nextid=1661089187920291335
https://groups.io/g/freertr/topic/rare_users_about_the_dpdk/93231306?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,100,93231306,previd%3D1661412819560986972,nextid%3D1661089187920291335&previd=1661412819560986972&nextid=1661089187920291335

long story short, i find it pretty hard to guess the right extra parameters:
a single core usually can easily keep some gigabit ports,
but a tengig interface usually will required a dedicated rx and tx core...
so without doing extensive nic detection, one cannot automate this properly,
and i can easily think about corner cases when further splitting an interface
could be necessary, so instead of doing this, an user configurable extra
parameter list is the way to go imho...
but good news that nor p4emu not p4xdp does not need these at all... :)

br,
cs