Skip to Content.
Sympa Menu

rare-dev - Re: [rare-dev] bf_fwd issues

Subject: Rare project developers

List archive

Re: [rare-dev] bf_fwd issues


Chronological Thread 
  • From: mc36 <>
  • To: , Frédéric LOUI <>
  • Cc: Alexander Gall <>, Vágó Tibor <>
  • Subject: Re: [rare-dev] bf_fwd issues
  • Date: Fri, 6 May 2022 18:32:33 +0200

hi,
well, bfd offload is not just about packet generation or replying to some
select packets...
after one detects that an issue happened to a neighbor, one have to
recalculate the whole
rib to exclude that interface from the forwarding... this is easily feasible
in freerouter
to do the trick... however the dataplane also have some work then here: one
have to walk
through the lpm, mpls, etc tables, and everywhere you find nexthopid=1234,
you have
to switch that to nexthopid=4321 or so... and exactly this is what is the
missing
piece for us to have bfd offload... moreover, receiving a bfd.req also
silences
us because our reply will keep the remote happy, and we're happy already since
we heard from the remote... and finally, we're experimenting with 4k
ppp-over-l2tp,
and freerouter have 5000ms default keepalive for ppp, plus we also run l2tp
zlb-ack,
plus the same number of ebgps, plus the tcp threads for each... all in all,
and it
keeps the sessions easily around 20% cpu utilization so i dont think that
there would
be an issue with a bfd for them too...
br,
cs



On 5/6/22 18:17, Fr d ric LOUI wrote:
Hi,

This reminds me that we started BFD hardware offload with TOFINO.
However, Dataplane packet generation trigger is limited.

It seems that in TOFINO2 this limitation has been lifted: (slide 5 and 6)
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjFoaXY3bj3AhVGrxoKHTc2B3cQFnoECAMQAQ&url=https%3A%2F%2Fopennetworking.org%2Fwp-content%2Fuploads%2F2020%2F04%2FJK-Lee-Slide-Deck.pdf&usg=AOvVaw2D4UEfrvUKwzwYxayyqtbA

It is maybe a good reason to resume BFD hardware offload with TOFINO1.

All the best,
Frederic

Le 6 mai 2022 16:53, mc36 <> a crit :

hi,

good points and thanks for taking a look on the dump.... yeahh, that
annotation logic is annoying...


since i wrote the previous mail, two things happened...

first, i reordered the tables, that doubled the performance:
https://github.com/rare-freertr/freeRtr/commit/0c52a8aef1028a9965ad28641e92dc6e479ded6e
(this also happened to the dpdk this week when we measured the 10g (c)pe for
performance)

next, i commented out (as per frederic idea) the from-hw in bf_ports:
https://github.com/rare-freertr/freeRtr/commit/f4a02a9b40fd8b976b63776a228619ff4f1f99d1

with these two, finally we're capable of picking up them all in a minute or
so, which is acceptable, imho....

ps: soo, regarding the table sizes, well, the nexthop table surely
accomodates the #cpe entries,
but, the same is true to the lpm table, where their unique /32 will appear,
moreover, since it's
a inter-as-option-b lns setup, the mpls table also have #cpe+#nrenigp/32s
labels in it...
(that latter is also goes to sram like the tunnel table does, since both have
only exact matches...)


mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:38 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 88509 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 42467 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.20/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:39 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 63933 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 54302 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.61/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:40 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 47549 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 60104 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.50/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:41 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 31165 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 66010 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.63/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:42 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 14781 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 76258 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (3.40/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:42 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 80983 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 0 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED off (0.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:43 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 75023 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 15130 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.07/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:44 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 58639 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 28053 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.68/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:45 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 42255 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 55038 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.76/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:46 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 34063 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 83646 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:46 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 17679 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 123106 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.20/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:48 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 1295 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 192690 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (3.68/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:48 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 133148 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.86/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:49 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 157842 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.01/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:50 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 160048 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.10/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:51 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 96458 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.16/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:53 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 105311 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.78/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:53 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 112770 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.99/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:54 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 118249 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.22/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:55 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 66270 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.12/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:56 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 98437 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.75/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:57 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 105061 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.77/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:57 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 133922 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:58 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 134228 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.25/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:28:59 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 8192 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 181498 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.28/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:00 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 115962 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.28/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:00 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 116906 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.53/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:01 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 121992 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.61/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:02 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 121992 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.80/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:03 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 16384 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 126979 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.01/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:03 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 65536 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 64247 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.11/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:04 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 57344 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 65825 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.71/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:05 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 69727 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.70/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:06 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 32768 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 70339 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.85/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:07 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 24576 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 86710 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.00/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:08 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 8192 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 88853 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (2.86/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:10 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 49152 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 60447 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.45/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:11 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40960 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 63331 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.44/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:12 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 73088 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 25900 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.23/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:12 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 64896 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 25900 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.38/0/1)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:13 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 56704 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 26847 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (1.42/0/0)
mc36@stordis:~$ date ; netstat -ano | grep 9080
Fri 06 May 2022 04:29:14 PM CEST
tcp 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN
off (0.00/0/0)
tcp 40320 0 127.0.0.1:52650 127.0.0.1:9080
ESTABLISHED off (0.00/0/0)
tcp 0 34579 127.0.0.1:9080 127.0.0.1:52650
ESTABLISHED probe (0.52/0/0)
mc36@stordis:~$




On 5/6/22 16:01, Alexander Gall wrote:
On Fri, 6 May 2022 15:20:18 +0200, mc36 <> said:
so here is the dump without the cumtime < 10...
as you can see, the l2tp4 message takes 10x the label4, labroute4, etc
messages summed up soo there must be something...
any python experts here who can spot the issue with that particular one?
My first impression is that it probably has to do with the relatively
large data size of the tbl_nexthop table for l2tpv4, possibly made
worse by the presence of field annotations. The time appears to be
spent mostly in the _Annotation.__eq__, _verify_size_and_set and
_convert_to_bytearray functions. It could well be that this is an
inefficiency in client.py, i.e. it would need to be addressed by
Intel, eventually.
Just a shot from the hip, could be completely wrong :)




Archive powered by MHonArc 2.6.19.

Top of Page