Skip to Content.
Sympa Menu

rare-dev - Re: [rare-dev] bf_fwd issues

Subject: Rare project developers

List archive

Re: [rare-dev] bf_fwd issues


Chronological Thread 
    < Chronological >      < Thread >
  • From: mc36 <>
  • To: , Alexander Gall <>
  • Cc: Vágó Tibor <>, Frédéric LOUI <>
  • Subject: Re: [rare-dev] bf_fwd issues
  • Date: Fri, 6 May 2022 16:53:43 +0200
hi,

good points and thanks for taking a look on the dump.... yeahh, that
annotation logic is annoying...


since i wrote the previous mail, two things happened...

first, i reordered the tables, that doubled the performance:
https://github.com/rare-freertr/freeRtr/commit/0c52a8aef1028a9965ad28641e92dc6e479ded6e
(this also happened to the dpdk this week when we measured the 10g (c)pe for
performance)

next, i commented out (as per frederic idea) the from-hw in bf_ports:
https://github.com/rare-freertr/freeRtr/commit/f4a02a9b40fd8b976b63776a228619ff4f1f99d1

with these two, finally we're capable of picking up them all in a minute or
so, which is acceptable, imho....

ps: soo, regarding the table sizes, well, the nexthop table surely
accomodates the #cpe entries,
but, the same is true to the lpm table, where their unique /32 will appear,
moreover, since it's
a inter-as-option-b lns setup, the mpls table also have #cpe+#nrenigp/32s
labels in it...
(that latter is also goes to sram like the tunnel table does, since both have
only exact matches...)


mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:38 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 88509 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 42467 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.20/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:39 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 63933 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 54302 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.61/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:40 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 47549 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 60104 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.50/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:41 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 31165 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 66010 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.63/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:42 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 14781 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 76258 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (3.40/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:42 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 80983 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 0 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED off (0.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:43 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 75023 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 15130 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.07/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:44 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 58639 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 28053 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.68/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:45 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 42255 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 55038 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.76/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:46 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 34063 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 83646 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:46 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 17679 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 123106 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.20/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:48 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 1295 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 192690 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (3.68/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:48 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 133148 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.86/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:49 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 157842 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.01/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:50 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 160048 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.10/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:51 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 96458 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.16/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:53 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 105311 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.78/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:53 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 112770 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.99/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:54 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 118249 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.22/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:55 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 66270 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.12/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:56 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 98437 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.75/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:57 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 105061 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.77/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:57 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 133922 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:58 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 134228 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.25/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:59 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 8192 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 181498 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.28/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:00 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 115962 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.28/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:00 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 116906 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.53/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:01 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 121992 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.61/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:02 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 121992 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.80/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:03 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 126979 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.01/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:03 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 64247 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.11/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:04 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 65825 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.71/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:05 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 69727 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.70/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:06 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 70339 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.85/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:07 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 86710 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:08 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 8192 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 88853 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (2.86/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:10 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 60447 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.45/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:11 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 63331 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.44/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:12 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 73088 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 25900 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.23/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:12 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 64896 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 25900 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.38/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:13 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 56704 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 26847 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.42/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:14 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40320 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 34579 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.52/0/0)
mc36@stordis:~$




On 5/6/22 16:01, Alexander Gall wrote:
On Fri, 6 May 2022 15:20:18 +0200, mc36 <> said:

so here is the dump without the cumtime < 10...
as you can see, the l2tp4 message takes 10x the label4, labroute4, etc
messages summed up soo there must be something...
any python experts here who can spot the issue with that particular one?

My first impression is that it probably has to do with the relatively
large data size of the tbl_nexthop table for l2tpv4, possibly made
worse by the presence of field annotations. The time appears to be
spent mostly in the _Annotation.__eq__, _verify_size_and_set and
_convert_to_bytearray functions. It could well be that this is an
inefficiency in client.py, i.e. it would need to be addressed by
Intel, eventually.

Just a shot from the hip, could be completely wrong :)


Archive powered by MHonArc 2.6.19.

Top of Page