Skip to Content.
Sympa Menu

rare-dev - Re: [rare-dev] bf_fwd issues

Subject: Rare project developers

List archive

Re: [rare-dev] bf_fwd issues


Chronological Thread 
    < Chronological >      < Thread >
  • From: Frédéric LOUI <>
  • To: cs <>
  • Cc: , Alexander Gall <>, Vágó Tibor <>
  • Subject: Re: [rare-dev] bf_fwd issues
  • Date: Fri, 6 May 2022 18:17:00 +0200
  • Dkim-filter: OpenDKIM Filter v2.10.3 zmtaauth02.partage.renater.fr 993DDA00E7
Hi,

This reminds me that we started BFD hardware offload with TOFINO.
However, Dataplane packet generation trigger is limited.

It seems that in TOFINO2 this limitation has been lifted: (slide 5 and 6)
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjFoaXY3bj3AhVGrxoKHTc2B3cQFnoECAMQAQ&url=https%3A%2F%2Fopennetworking.org%2Fwp-content%2Fuploads%2F2020%2F04%2FJK-Lee-Slide-Deck.pdf&usg=AOvVaw2D4UEfrvUKwzwYxayyqtbA

It is maybe a good reason to resume BFD hardware offload with TOFINO1.

All the best,
Frederic

> Le 6 mai 2022 à 16:53, mc36 <> a écrit :
>
> hi,
>
> good points and thanks for taking a look on the dump.... yeahh, that
> annotation logic is annoying...
>
>
> since i wrote the previous mail, two things happened...
>
> first, i reordered the tables, that doubled the performance:
> https://github.com/rare-freertr/freeRtr/commit/0c52a8aef1028a9965ad28641e92dc6e479ded6e
> (this also happened to the dpdk this week when we measured the 10g (c)pe
> for performance)
>
> next, i commented out (as per frederic idea) the from-hw in bf_ports:
> https://github.com/rare-freertr/freeRtr/commit/f4a02a9b40fd8b976b63776a228619ff4f1f99d1
>
> with these two, finally we're capable of picking up them all in a minute or
> so, which is acceptable, imho....
>
> ps: soo, regarding the table sizes, well, the nexthop table surely
> accomodates the #cpe entries,
> but, the same is true to the lpm table, where their unique /32 will appear,
> moreover, since it's
> a inter-as-option-b lns setup, the mpls table also have #cpe+#nrenigp/32s
> labels in it...
> (that latter is also goes to sram like the tunnel table does, since both
> have only exact matches...)
>
>
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:38 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 88509 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 42467 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.20/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:39 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 63933 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 54302 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.61/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:40 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 47549 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 60104 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.50/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:41 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 31165 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 66010 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.63/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:42 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 14781 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 76258 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (3.40/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:42 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 80983 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 0 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED off (0.00/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:43 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 75023 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 15130 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.07/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:44 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 58639 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 28053 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.68/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:45 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 42255 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 55038 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.76/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:46 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 34063 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 83646 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.00/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:46 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 17679 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 123106 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.20/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:48 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 1295 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 192690 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (3.68/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:48 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 133148 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.86/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:49 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 157842 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.01/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:50 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 160048 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.10/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:51 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 96458 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.16/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:53 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 105311 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.78/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:53 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 112770 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.99/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:54 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 118249 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.22/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:55 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 66270 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.12/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:56 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 98437 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.75/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:57 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 105061 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.77/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:57 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 133922 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.00/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:58 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 134228 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.25/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:28:59 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 8192 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 181498 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.28/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:00 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 115962 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.28/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:00 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 116906 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.53/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:01 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 121992 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.61/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:02 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 121992 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.80/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:03 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 126979 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.01/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:03 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 64247 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.11/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:04 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 65825 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.71/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:05 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 69727 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.70/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:06 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 70339 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.85/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:07 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 86710 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.00/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:08 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 8192 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 88853 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (2.86/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:10 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 60447 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.45/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:11 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 63331 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.44/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:12 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 73088 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 25900 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.23/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:12 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 64896 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 25900 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.38/0/1)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:13 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 56704 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 26847 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (1.42/0/0)
> mc36@stordis:~$ date ; netstat -ano | grep 9080
> Fri 06 May 2022 04:29:14 PM CEST
> tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
> off (0.00/0/0)
> tcp 40320 0 127.0.0.1:52650 127.0.0.1:9080
> ESTABLISHED off (0.00/0/0)
> tcp 0 34579 127.0.0.1:9080 127.0.0.1:52650
> ESTABLISHED probe (0.52/0/0)
> mc36@stordis:~$
>
>
>
>
> On 5/6/22 16:01, Alexander Gall wrote:
>> On Fri, 6 May 2022 15:20:18 +0200, mc36 <> said:
>>> so here is the dump without the cumtime < 10...
>>> as you can see, the l2tp4 message takes 10x the label4, labroute4, etc
>>> messages summed up soo there must be something...
>>> any python experts here who can spot the issue with that particular one?
>> My first impression is that it probably has to do with the relatively
>> large data size of the tbl_nexthop table for l2tpv4, possibly made
>> worse by the presence of field annotations. The time appears to be
>> spent mostly in the _Annotation.__eq__, _verify_size_and_set and
>> _convert_to_bytearray functions. It could well be that this is an
>> inefficiency in client.py, i.e. it would need to be addressed by
>> Intel, eventually.
>> Just a shot from the hip, could be completely wrong :)


Archive powered by MHonArc 2.6.19.

Top of Page