RARE user and assistance email list

This picture is I think before I pinned my benchmark to cpu cores. I used all the tweaks

I connected my two boxes with a x710-da4 (each 4x10g) straight to each other, added four ips on freertr config. 

BIOS: Enable HT, disable powersaving C3/C6, enable vtd

You want to check your numactl -H, memory 'architecture', pci-e speeds (lspci -vv), add hugepages parameters to bootkernel, iommu flags for certain drivers

I only have 6 real CPU cores shared with the NUMA zone 1 on my box (second cpu socket). 4 ifaces x (1 RX core + 1 TX core)+ 2n fwd cores makes this is tough spot, so I had to oversubscribe, I think this is responsible for a large part of my variance in testing right now. Same problem applies for pinning benchmarks to cores and interface count vs corecount. I did my best with what I have and this largely resolved variance in performance.

I'm looking into getting minimum 8 cores some time soon. Input welcome, e5v3 or v4 let me know what to get/is cheap ebay/aliexpress (2650v3 for test and 2698v4 for freertr?)

My super sweet basic bash test suite, I need to wire variables more thoroughly through it so eg. the mtu tests works and more script arguments

root@test:~# cat run.sh

#!/bin/bash

set -e

./setup.sh

./validate.sh

./test.sh | awk '/SUM/{ print $0; sum+=$6; t=$2; dsum+=$4; dunit=$5; unit=$7  } END { print "[AGG]\t"t"  sec\t",dsum,dunit,sum,unit }'

./clean.sh

root@test:~# cat setup.sh

#!/bin/bash

 

mtu=1500

 

ip netns add p0

ip netns add p1

ip netns add p2

ip netns add p3

 

ip link set dev enp129s0f0 netns p0

ip link set dev enp129s0f1 netns p1

ip link set dev enp129s0f2 netns p2

ip link set dev enp129s0f3 netns p3

 

ip netns exec p0 ip a add 192.168.19.20/24 dev enp129s0f0

ip netns exec p1 ip a add 192.168.20.20/24 dev enp129s0f1

ip netns exec p2 ip a add 192.168.21.20/24 dev enp129s0f2

ip netns exec p3 ip a add 192.168.22.20/24 dev enp129s0f3

 

ip netns exec p0 ip l set mtu $mtu dev enp129s0f0

ip netns exec p1 ip l set mtu $mtu dev enp129s0f1

ip netns exec p2 ip l set mtu $mtu dev enp129s0f2

ip netns exec p3 ip l set mtu $mtu dev enp129s0f3

 

ip netns exec p0 ip l set up dev enp129s0f0

ip netns exec p1 ip l set up dev enp129s0f1

ip netns exec p2 ip l set up dev enp129s0f2

ip netns exec p3 ip l set up dev enp129s0f3

 

ip netns exec p0 ip route add 0.0.0.0/0 via 192.168.19.19

ip netns exec p1 ip route add 0.0.0.0/0 via 192.168.20.19

ip netns exec p2 ip route add 0.0.0.0/0 via 192.168.21.19

ip netns exec p3 ip route add 0.0.0.0/0 via 192.168.22.19

 

root@test:~# cat validate.sh

#!/bin/bash

set -e

ip netns exec p0 ping 192.168.19.19 -q -c 1 -w 1 &>/dev/null

ip netns exec p1 ping 192.168.20.19 -q -c 1 -w 1 &>/dev/null

ip netns exec p2 ping 192.168.21.19 -q -c 1 -w 1 &>/dev/null

ip netns exec p3 ping 192.168.22.19 -q -c 1 -w 1 &>/dev/null

root@test:~# cat test.sh

#!/bin/bash

 

t=15

O=5

P=4

 

ip netns exec p0 taskset -c 6 iperf3 -s -1 &>/dev/null &

ip netns exec p1 taskset -c 7 iperf3 -s -1 &>/dev/null &

ip netns exec p2 taskset -c 8 iperf3 -s -1 &>/dev/null &

ip netns exec p3 taskset -c 9 iperf3 -s -1 &>/dev/null &

 

sleep 3

 

ip netns exec p0 taskset -c 2 iperf3 -c 192.168.20.20 -t $t -i 0 --forceflush -P $P -O $O | grep sender &

ip netns exec p1 taskset -c 3 iperf3 -c 192.168.21.20 -t $t -i 0 --forceflush -P $P -O $O | grep sender &

ip netns exec p2 taskset -c 4 iperf3 -c 192.168.22.20 -t $t -i 0 --forceflush -P $P -O $O | grep sender &

ip netns exec p3 taskset -c 5 iperf3 -c 192.168.19.20 -t $t -i 0 --forceflush -P $P -O $O | grep sender

 

I used  -2 8192 and set the mtu to jumbo in rtr-sw (+ vethxB as well, not sure if needed, I think your install.sh/bootscript does some mtu/offload magic upon start).
got a nearly perfect (9/10) 36.5Gbits (32G was the outlier). When I then kept this on freertr sides but lowered the test mtu back to 1500 the results became more stable hovering around 34gpbs with 2gpbs variance.

So what do you think, I also discovered my x710-da4 is slightly limited by pci-e v3 x8 BW (7880MB/s) so that might potentially not help achieve wire-rates :) I have enough 10gpbs adapters to work around it and use my numa zone 0 but that would be too easy ;) I'm trying to load 25gpbs adapters, but those mlnx4/5 ebay cards look very tempting

ow, about cpu load; when you jumbo frame it it doesnt uses more than 25pct, without it reaches 60pct.

_._,_._,_

---

   Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#170) | | | Mute This Topic | New Topic
Your Subscription | | Unsubscribe []

_._,_._,_