Skip to Content.

rare-users - [gn4-3-wp6-t1-wb-RARE] p4dpdk performance

Subject: RARE user and assistance email list

List archive


[gn4-3-wp6-t1-wb-RARE] p4dpdk performance


Chronological Thread 
    < Chronological >      < Thread >
  • From: mc36 <>
  • To: ,
  • Subject: [gn4-3-wp6-t1-wb-RARE] p4dpdk performance
  • Date: Thu, 17 Sep 2020 14:47:17 +0200
hi,

so as noted yesterday, i got my 10gbps nics finally so i had a quick
experiment in my homenet.
initially i hooked the traffic up with my stordis but after the 2nd copy of
my rib+fib it given
up and made a permanent forwarding loop so i removed that guy and continued
with a direct loop...
the idea is that i used point2point vlans where i enabled my igp in different
vrfs on the ends and this made up a chain:
sdn6.101--sdn7.101--sdn7.102---sdn6.102.....
my rib have 250 entries (per vrf) that is 2.5k but my fib uses sr with block
size 300
so i ended up with an 3.7k fib programmed in total...

the forwarding was done by a 7 years old i7-3770 over intel nics.
finally i reached a constant 6+6=12gbps rate with 30% load of 2
threads dealing with the 2 tengig ports, and a 9% cpu for the gig
port's thread... those tengig ports are unfortunately bound to the
threads of the same core so it could be better, moreover if i had
more cores then i would bound rx and tx workers to different cores...
finally i given up the vrf/subif/igp creation, we can estimate
from these numbers enough: one thread forwards 6gbps with 30% load,
so one thread could do 18gbps with 90% load.... and as noted above,
we can split rx and tx so it's quite inspiring!!!!

afterwards, i've enabled macsec on each subifs... at this time
i as able to maintain 1.2gbps flow / port so i had a 2.4gbps total
crypto throughput with 2 threads maxed out at 100%, but that cpu
have 4 cores (8 threads0 so that cpu is capable of 9.6gbps total
crypto throughput. i used aes256+sha1 in this experiment.

regards,
cs



mchome#show interfaces sd6 hwhistory
6865m|
6178m| # # ## # # # # #
5492m|#### # ### # ## #### ## # # # ####### # #### ##
4805m|########### # ### #####################################
4119m|########### ###############################################
3432m|###########################################################
2746m|###########################################################
2059m|###########################################################
1373m|###########################################################
686m|###########################################################
0|###########################################################
bps|0---------10--------20--------30--------40--------50--------
seconds

13g|
12g| **** ** * **
10g| ************** *
9341m|*************** ********** **
8007m|*************** ************** * * ** *
6672m|*************** ************** **********
5338m|##############* ************** ********** * *
4003m|############### *#############*********** ***
2669m|############### *#############*########## ***
1334m|############### *#############*########## ##*
0|###########################################################
bps|0---------10--------20--------30--------40--------50--------
minutes

10g|
9706m|*
8628m|*
7549m|*
6471m|*
5392m|*
4314m|*
3235m|*
2157m|*
1078m|*
0|#
bps|0---------10--------20--------30--------40--------50--------
hours


mchome#show interfaces sd7 hwhistory
6865m|
6178m| # # # ## # # # #
5492m|####### # ### # ## #### ## # # # ####### # ####
4805m|############## # ### ##################################
4119m|############## ############################################
3432m|###########################################################
2746m|###########################################################
2059m|###########################################################
1373m|###########################################################
686m|###########################################################
0|###########################################################
bps|0---------10--------20--------30--------40--------50--------
seconds

13g|
12g| **** ** * **
10g| ************** *
9341m|*************** ********** **
8006m|*************** ************** * * ** *
6672m|*************** ************** **********
5337m|##############* ************** ********** * *
4003m|############### *#############*********** ***
2668m|############### *#############*########## ***
1334m|############### *#############*########## ##*
0|###########################################################
bps|0---------10--------20--------30--------40--------50--------
minutes

10g|
9705m|*
8627m|*
7549m|*
6470m|*
5392m|*
4313m|*
3235m|*
2156m|*
1078m|*
0|##
bps|0---------10--------20--------30--------40--------50--------
hours


mchome#show interfaces bun1 hwhistory
686m|
618m| # # # ## # # # #
549m| ######## # ### # ## #### ## # # # ####### # ##
480m| ############### # ### ################################
412m|################ ##########################################
343m|###########################################################
274m|###########################################################
206m|###########################################################
137m|###########################################################
68m|###########################################################
0|###########################################################
bps|0---------10--------20--------30--------40--------50--------
seconds

1415m|
1274m| * * * * * * * * * ******** * *
1132m| ************** *********** ** ********** ***
990m|*************** ************** ********** ***
849m|*************** ************** ********** ***
707m|*************** ************** ********** ***
566m|**********#*#** *#############*#########* *#*
424m|############### *#############*########## *#*
283m|############### *#############*########## ##*
141m|############### *#############*########## ##*
0|###########################################################
bps|0---------10--------20--------30--------40--------50--------
minutes

1415m|
1274m|*
1132m|*
990m|*
849m|*
707m|**
566m|**
424m|**
283m|**
141m|#* *
0|####
bps|0---------10--------20--------30--------40--------50--------
hours


mchome#


mchome#show mpls forwarding | count
3678 lines, 22116 words, 130134 characters

mchome#show ipv4 route inet | count
230 lines, 1413 words, 11876 characters

mchome#
mchome#
mchome#show config-differences
server p4lang p4
export-vrf v2 11
export-vrf v3 12
export-vrf v4 13
export-vrf v5 14
export-vrf v6 15
export-vrf v7 16
export-vrf v8 17
export-vrf v9 18
export-vrf v10 19
export-vrf v11 20
export-port sdn7.101 101 0
export-port sdn7.102 102 0
export-port sdn7.103 103 0
export-port sdn7.104 104 0
export-port sdn7.105 105 0
export-port sdn7.106 106 0
export-port sdn7.107 107 0
export-port sdn7.108 108 0
export-port sdn7.109 109 0
export-port sdn7.110 110 0
export-port sdn6.101 201 0
export-port sdn6.102 202 0
export-port sdn6.103 203 0
export-port sdn6.104 204 0
export-port sdn6.105 205 0
export-port sdn6.106 206 0
export-port sdn6.107 207 0
export-port sdn6.108 208 0
export-port sdn6.109 209 0
export-port sdn6.110 210 0
exit
interface bundle1.182
no template template1
no ipv6 address 2001:db8:44::1 ffff:ffff:ffff:ffff::
no vrf forwarding inet
lldp enable
cdp enable
vrf forwarding v11
mpls enable
router lsrp4 11 enable
no log-link-change
exit
vrf definition v10
rd 1:10
exit
vrf definition v11
rd 1:11
exit
vrf definition v2
rd 1:2
exit
vrf definition v3
rd 1:3
exit
vrf definition v4
rd 1:4
exit
vrf definition v5
rd 1:5
exit
vrf definition v6
rd 1:6
exit
vrf definition v7
rd 1:7
exit
vrf definition v8
rd 1:8
exit
vrf definition v9
rd 1:9
exit
router lsrp4 2
vrf v2
router-id 3.3.3.2
segrout 300 242
redistribute connected
exit
router lsrp4 3
vrf v3
router-id 3.3.3.3
segrout 300 244
redistribute connected
exit
router lsrp4 4
vrf v4
router-id 2.2.2.4
segrout 300 241
redistribute connected
exit
router lsrp4 5
vrf v5
router-id 2.2.2.5
segrout 300 243
redistribute connected
exit
router lsrp4 6
vrf v6
router-id 3.3.3.6
segrout 300 245
redistribute connected
exit
router lsrp4 7
vrf v7
router-id 2.2.2.7
segrout 300 246
redistribute connected
exit
router lsrp4 8
vrf v8
router-id 3.3.3.8
segrout 300 247
redistribute connected
exit
router lsrp4 9
vrf v9
router-id 2.2.2.9
segrout 300 248
redistribute connected
exit
router lsrp4 10
vrf v10
router-id 3.3.3.10
segrout 300 249
redistribute connected
exit
router lsrp4 11
vrf v11
router-id 2.2.2.11
segrout 300 250
redistribute connected
exit
interface sdn6.101
no description
lldp enable
cdp enable
vrf forwarding v4
ipv4 address 2.2.2.1 255.255.255.252
mpls enable
router lsrp4 4 enable
no shutdown
no log-link-change
exit
interface sdn6.102
no description
lldp enable
cdp enable
vrf forwarding v4
ipv4 address 2.2.2.5 255.255.255.252
mpls enable
router lsrp4 4 enable
no shutdown
no log-link-change
exit
interface sdn6.103
no description
lldp enable
cdp enable
vrf forwarding v5
ipv4 address 2.2.2.10 255.255.255.252
mpls enable
router lsrp4 5 enable
no shutdown
no log-link-change
exit
interface sdn6.104
no description
lldp enable
cdp enable
vrf forwarding v5
ipv4 address 2.2.2.13 255.255.255.252
mpls enable
router lsrp4 5 enable
no shutdown
no log-link-change
exit
interface sdn6.105
no description
lldp enable
cdp enable
vrf forwarding v6
ipv4 address 2.2.2.18 255.255.255.252
mpls enable
router lsrp4 6 enable
no shutdown
no log-link-change
exit
interface sdn6.106
no description
lldp enable
cdp enable
vrf forwarding v6
ipv4 address 2.2.2.21 255.255.255.252
mpls enable
router lsrp4 6 enable
no shutdown
no log-link-change
exit
interface sdn6.107
no description
lldp enable
cdp enable
vrf forwarding v8
ipv4 address 2.2.2.25 255.255.255.252
mpls enable
router lsrp4 8 enable
no shutdown
no log-link-change
exit
interface sdn6.108
no description
lldp enable
cdp enable
vrf forwarding v8
ipv4 address 2.2.2.29 255.255.255.252
mpls enable
router lsrp4 8 enable
no shutdown
no log-link-change
exit
interface sdn6.109
no description
lldp enable
cdp enable
vrf forwarding v10
ipv4 address 2.2.2.33 255.255.255.252
mpls enable
router lsrp4 10 enable
no shutdown
no log-link-change
exit
interface sdn6.110
no description
lldp enable
cdp enable
vrf forwarding v10
ipv4 address 2.2.2.37 255.255.255.252
mpls enable
router lsrp4 10 enable
no shutdown
no log-link-change
exit
interface sdn7.101
no description
vrf forwarding inet
ipv4 address 2.2.2.2 255.255.255.252
template template1
no shutdown
exit
interface sdn7.102
no description
lldp enable
cdp enable
vrf forwarding v2
ipv4 address 2.2.2.6 255.255.255.252
mpls enable
router lsrp4 2 enable
no shutdown
no log-link-change
exit
interface sdn7.103
no description
lldp enable
cdp enable
vrf forwarding v2
ipv4 address 2.2.2.9 255.255.255.252
mpls enable
router lsrp4 2 enable
no shutdown
no log-link-change
exit
interface sdn7.104
no description
lldp enable
cdp enable
vrf forwarding v3
ipv4 address 2.2.2.14 255.255.255.252
mpls enable
router lsrp4 3 enable
no shutdown
no log-link-change
exit
interface sdn7.105
no description
lldp enable
cdp enable
vrf forwarding v3
ipv4 address 2.2.2.17 255.255.255.252
mpls enable
router lsrp4 3 enable
no shutdown
no log-link-change
exit
interface sdn7.106
no description
lldp enable
cdp enable
vrf forwarding v7
ipv4 address 2.2.2.22 255.255.255.252
mpls enable
router lsrp4 7 enable
no shutdown
no log-link-change
exit
interface sdn7.107
no description
lldp enable
cdp enable
vrf forwarding v7
ipv4 address 2.2.2.26 255.255.255.252
mpls enable
router lsrp4 7 enable
no shutdown
no log-link-change
exit
interface sdn7.108
no description
lldp enable
cdp enable
vrf forwarding v9
ipv4 address 2.2.2.30 255.255.255.252
mpls enable
router lsrp4 9 enable
no shutdown
no log-link-change
exit
interface sdn7.109
no description
lldp enable
cdp enable
vrf forwarding v9
ipv4 address 2.2.2.34 255.255.255.252
mpls enable
router lsrp4 9 enable
no shutdown
no log-link-change
exit
interface sdn7.110
no description
lldp enable
cdp enable
vrf forwarding v11
ipv4 address 2.2.2.38 255.255.255.252
mpls enable
router lsrp4 11 enable
no shutdown
no log-link-change
exit

mchome#







acer#traceroute 10.10.10.199 /delay 22
tracing 10.10.10.199, src=10.10.10.11, vrf=inet, prt=0/33440, tim=1000,
tos=0, len=64
1 10.10.10.11 time=0
2 2.2.2.2 time=4, mpls=692938
3 2.2.2.5 time=7, mpls=275058
4 2.2.2.9 time=5, mpls=64282
5 2.2.2.13 time=4, mpls=869480
6 2.2.2.17 time=5, mpls=57415
7 2.2.2.21 time=3, mpls=983162
8 2.2.2.26 time=4, mpls=696635
9 2.2.2.29 time=4, mpls=442815
10 2.2.2.34 time=3, mpls=1013296
11 2.2.2.37 time=5, mpls=99515
12 10.1.1.126 time=3, mpls=468753
13 10.10.10.199 time=3, mpls=940656
acer#






top - 14:27:37 up 1 day, 1:54, 1 user, load average: 1.27, 1.26, 1.27
Tasks: 171 total, 2 running, 169 sleeping, 0 stopped, 0 zombie
%Cpu0 : 3.8 us, 1.4 sy, 0.0 ni, 94.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0
st
%Cpu1 : 7.9 us, 1.7 sy, 0.0 ni, 90.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu2 : 8.6 us, 2.7 sy, 0.0 ni, 88.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu3 : 2.7 us, 2.7 sy, 0.0 ni, 94.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu4 : 5.5 us, 2.1 sy, 0.0 ni, 92.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu5 : 5.5 us, 2.7 sy, 0.0 ni, 91.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu6 : 27.1 us, 1.8 sy, 0.0 ni, 71.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu7 : 28.8 us, 1.1 sy, 0.0 ni, 70.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
MiB Mem : 11854.4 total, 7313.8 free, 3032.4 used, 1508.1 buff/cache
MiB Swap: 7168.0 total, 7168.0 free, 0.0 used. 8523.6 avail Mem



top - 14:45:32 up 1 day, 2:12, 1 user, load average: 3.98, 2.25, 1.62
Tasks: 171 total, 1 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu0 : 2.0 us, 1.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu1 : 1.0 us, 0.3 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0
st
%Cpu2 : 18.6 us, 2.4 sy, 0.0 ni, 79.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu3 : 4.5 us, 1.0 sy, 0.0 ni, 94.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu4 : 6.1 us, 1.0 sy, 0.0 ni, 92.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu5 : 1.7 us, 1.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu6 : 97.3 us, 2.3 sy, 0.0 ni, 0.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
%Cpu7 : 81.3 us, 2.0 sy, 0.0 ni, 16.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
MiB Mem : 11854.4 total, 7299.9 free, 3045.4 used, 1509.1 buff/cache
MiB Swap: 7168.0 total, 7168.0 free, 0.0 used. 8510.4 avail Mem






[ 3] 987.0-988.0 sec 67.6 MBytes 567 Mbits/sec
[ 3] 988.0-989.0 sec 62.2 MBytes 522 Mbits/sec
[ 3] 989.0-990.0 sec 63.9 MBytes 536 Mbits/sec
[ 3] 990.0-991.0 sec 58.8 MBytes 493 Mbits/sec
[ 3] 991.0-992.0 sec 59.9 MBytes 502 Mbits/sec
[ 3] 992.0-993.0 sec 72.1 MBytes 605 Mbits/sec
[ 3] 993.0-994.0 sec 70.0 MBytes 587 Mbits/sec
[ 3] 994.0-995.0 sec 67.9 MBytes 569 Mbits/sec
[ 3] 995.0-996.0 sec 70.4 MBytes 590 Mbits/sec
[ 3] 996.0-997.0 sec 58.9 MBytes 494 Mbits/sec
[ 3] 997.0-998.0 sec 69.6 MBytes 584 Mbits/sec
[ 3] 998.0-999.0 sec 61.0 MBytes 512 Mbits/sec
[ 3] 0.0-999.0 sec 59.3 GBytes 510 Mbits/sec



[ 4] 242.0-243.0 sec 12.0 MBytes 100 Mbits/sec
[ 4] 243.0-244.0 sec 12.0 MBytes 101 Mbits/sec
[ 4] 244.0-245.0 sec 12.3 MBytes 103 Mbits/sec
[ 4] 245.0-246.0 sec 12.0 MBytes 101 Mbits/sec
[ 4] 246.0-247.0 sec 11.9 MBytes 99.8 Mbits/sec
[ 4] 247.0-248.0 sec 12.3 MBytes 103 Mbits/sec
[ 4] 248.0-249.0 sec 11.9 MBytes 100 Mbits/sec
[ 4] 249.0-250.0 sec 12.0 MBytes 101 Mbits/sec
[ 4] 250.0-251.0 sec 11.9 MBytes 100 Mbits/sec
[ 4] 251.0-252.0 sec 12.1 MBytes 102 Mbits/sec
[ 4] 252.0-253.0 sec 12.0 MBytes 101 Mbits/sec


Archive powered by MHonArc 2.6.19.

Top of Page