RARE user and assistance email list

["David Schmitz" <>]

Hi Csaba,

On Wed, 7 Jun 2023, mc36 wrote:

> Date: Wed, 7 Jun 2023 06:11:47 +0200
> From: mc36 <>
> Reply-To: , 
> To: David Schmitz <>, 
> Subject: Re: [freertr] Issues with actual FlowSpec filtering,
>     especially for rules announced via exabgp (fwd)
>
> getting back to you, sooooo
>
> first of all, extend your topology with 1+ more freerouter.... then r1 will 
> only
>
> flowspec-advertise the others will only flowspec-install... and the rest of 
> the routers,
>
> will only install, as usual... in this case a route reflector will be needed, 
> that is
>
> where exabgp and the rest should be peered to...
>
>
>
> in the meanwhile i tryed in my nren's ibgp, where we have asr9ks and the 
> flowspec ruels comes from 2 arbor networks cleaners
>
> everything seems fineeee (see below) so my guess is it's exabgp's 
> formatter.... to see this a bit closer, please do the following:
>
> r1#packet capture eth1
>
> then start announcing from exabgp
> wait a bit
> stop exabgp
>
> r1#packet capture eth1
>
> gimme the resulting eth1.pcap plus
>
> show ipv4 bgp 1955 flowspec wireformat 601:1dc3:c7c7:2800::#::
find attached r1-eth2--exabgppeering.pcap
(eth2 is r1's tap interface to exabgp)
which includes exabgp BGP connection establishment and 
announcement of a FlowSpec rule
(exabgpcli announce flow route source-ipv4 10.1.10.1/32 destination-ipv4 
10.2.10.2/32 protocol =icmp)

rtr1#show ipv4 bgp 1 flowspec wireformat f01:200a:20a:202:200a:10a:103:8101#:: 
00000000: ff ff ff ff  ff ff ff ff  ff ff ff ff  ff ff ff ff 
00000010: 00 47 02 00  00 00 30 40  01 01 00 40  02 06 02 01 
00000020: 00 00 0b b9  40 05 04 00  00 00 64 80  0e 19 00 01 
00000030: 85 04 0a 03  0a 03 00 0f  01 20 0a 02  0a 02 02 20 
00000040: 0a 01 0a 01  03 81 01

rtr2#show ipv4 bgp 1 flowspec wireformat f01:200a:20a:202:200a:10a:103:8101#:: 
00000000: ff ff ff ff  ff ff ff ff  ff ff ff ff  ff ff ff ff 
00000010: 00 4b 02 00  00 00 34 40  01 01 00 40  02 0a 02 02 
00000020: 00 00 0f ab  00 00 0b b9  40 05 04 00  00 00 64 80 
00000030: 0e 19 00 01  85 04 0a 04  0a 0b 00 0f  01 20 0a 02 
00000040: 0a 02 02 20  0a 01 0a 01  03 81 01


This rule showed up on
rtr2 like this:

rtr2# show ipv4 bgp 1 flowspec summary
neighbor    as    learn  accept  will  done  uptime
10.4.10.11  4011  1      1       1     1     00:01:41

rtr2# show ipv4 flwspc CORE
prefix                                     hop         metric      aspath
f01:200a:20a:202:200a:10a:103:8101#:: 0:0  10.4.10.11  20/100/0/0  4011 3001

rtr2# show router bgp4 1 redisted flowspec
prefix  hop  metric  aspath

rtr2# show router bgp4 1 computed flowspec
prefix                                     hop         metric      aspath
f01:200a:20a:202:200a:10a:103:8101#:: 0:0  10.4.10.11  20/100/0/0  4011 3001

rtr2# show ipv4 bgp 1 flowspec database
prefix                                     hop         metric      aspath
f01:200a:20a:202:200a:10a:103:8101#:: 0:0  10.4.10.11  20/100/0/0  4011 3001

rtr2# show policy-map flowspec CORE ipv4
seq  chld  queue  intrvl  byt/int  rxb  rxp  trnsmt                      ace
1    0     0/128  100     0        0    0    tx=0(0) rx=0(0) drp=0(0)    1-1 
10.1.10.1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all 10.2.10.2 
ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
2    0     0/128  100     0        700  8    tx=700(8) rx=0(0) drp=0(0)

But it did not mitigate (from host1=10.1.10.1 to host2=10.2.10.2 
interconnected via rtr2)

host1#ping 10.2.10.2 vrf CORE 
pinging 10.2.10.2, src=null, vrf=CORE, cnt=5, len=64, df=false, tim=1000, gap=0, ttl=255, tos=0, sgt=0, flow=0, fill=0, alrt=-1, sweep=false, multi=false
..!!!
result=60.0%, recv/sent/lost/err=3/5/2/0, took 2050, min/avg/max/dev 
rtt=2/3.6/5/1.5, ttl 254/254/254/0.0, tos 0/0.0/0/0.0

host1#ping 10.2.10.2 vrf CORE 
pinging 10.2.10.2, src=null, vrf=CORE, cnt=5, len=64, df=false, tim=1000, gap=0, ttl=255, tos=0, sgt=0, flow=0, fill=0, alrt=-1, sweep=false, multi=false
!!!!!
result=100.0%, recv/sent/lost/err=5/5/0/0, took 13, min/avg/max/dev 
rtt=1/1.8/3/0.5, ttl 254/254/254/0.0, tos 0/0.0/0/0.0

rtr2# show policy-map flowspec CORE ipv4
seq  chld  queue  intrvl  byt/int  rxb   rxp  trnsmt                        
ace
1    0     0/128  100     0        576   9    tx=576(9) rx=0(0) drp=0(0)    
1-1 10.1.10.1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all 10.2.10.2 
ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
2    0     0/128  100     0        1490  22   tx=1490(22) rx=0(0) drp=0(0)

> where the the ipv6 alike address is the rule in the bgp rib that fails to 
> decode properly....
>
> ps: https://datatracker.ietf.org/doc/draft-hares-idr-flowspec-v2/ <---- this 
> is here with a reason...
Yes, in future we should see how to do more regarding newer FlowSpec 
standards.
For now, we are more or less tight to exabgp's support of only v1.

Best Regards
David

> br,
>
> cs
>
>
>
>
> rtr2.c4e#show ipv4 bgp 1955 summary
> neighbor        as    ready  learn  sent  uptime
> 195.111.97.91   1955  false  0      0     never
> 195.111.97.92   1955  true   32424  7     00:55:58
> 195.111.97.93   1955  true   32424  7     00:56:00
> 195.111.97.179  1955  true   32437  7     00:55:58
>
> rtr2.c4e#show ipv4 bgp 1955 flowspec summary
> neighbor        as    learn  accept  will  done  uptime
> 195.111.97.91   1955  0      0       0     0     never
> 195.111.97.92   1955  3      3       0     0     00:56:00
> 195.111.97.93   1955  3      3       0     0     00:56:02
> 195.111.97.179  1955  16     16      0     0     00:56:00
>
> rtr2.c4e#   rtr2.c4e#configure
> rtr2.c4e(cfg)#router bgp4 1955
> rtr2.c4e(cfg-rtr)#show ipv4 bgp 1955 flowspec database
> prefix                       hop             metric       aspath
> 601:1dc3:c7c7:2800::#:: 0:0  195.111.97.179  200/100/2/0
> 601:2054:ce32:ee00::#:: 0:0  195.111.97.179  200/100/2/0
> 601:2054:ce32:ef00::#:: 0:0  195.111.97.179  200/100/2/0
> 601:2054:ce32:f000::#:: 0:0  195.111.97.179  200/100/2/0
> 601:2054:ce7f:4d00::#:: 0:0  195.111.97.179  200/100/2/0
> 601:2054:ce7f:4e00::#:: 0:0  195.111.97.179  200/100/2/0
> 601:2054:ce82:f900::#:: 0:0  195.111.97.179  200/100/2/0
> 601:20c1:6f1:c300::#:: 0:0   195.111.97.179  200/100/2/0
> 601:20c1:6f1:c400::#:: 0:0   195.111.97.179  200/100/2/0
> 601:20c1:6f1:c500::#:: 0:0   195.111.97.179  200/100/2/0
> 601:20c1:e029:8600::#:: 0:0  195.111.97.179  200/100/2/0
> 601:20c1:e0a3:1d00::#:: 0:0  195.111.97.179  200/100/2/0
> 601:20c1:e10e:8d00::#:: 0:0  195.111.97.179  200/100/2/0
> 602:2058:da11:2100::#:: 0:0  195.111.97.92   200/100/0/0
> 602:2058:da11:2100::#:: 0:0  195.111.97.93   200/100/0/0
> 602:2058:da11:2100::#:: 0:0  195.111.97.179  200/100/0/0
> 602:209c:e63f:4600::#:: 0:0  195.111.97.92   200/100/0/0
> 602:209c:e63f:4600::#:: 0:0  195.111.97.93   200/100/0/0
> 602:209c:e63f:4600::#:: 0:0  195.111.97.179  200/100/0/0
> 602:20ca:3cf5:8200::#:: 0:0  195.111.97.92   200/100/0/0
> 602:20ca:3cf5:8200::#:: 0:0  195.111.97.93   200/100/0/0
> 602:20ca:3cf5:8200::#:: 0:0  195.111.97.179  200/100/0/0
>
> rtr2.c4e(cfg-rtr)#show policy-map flowspec inet ipv4
> % no such policy
> rtr2.c4e(cfg-rtr)#flowspec-install
> rtr2.c4e(cfg-rtr)#show policy-map flowspec inet ipv4
> seq  chld  queue  intrvl  byt/int  rxb   rxp  trnsmt 
> ace
> 1    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 195.199.199.40 ffff:ffff:ffff:ffff:ffff:ffff:ffff:fff8 all
> 2    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 84.206.50.238 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 3    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 84.206.50.239 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 4    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 84.206.50.240 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 5    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 84.206.127.77 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 6    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 84.206.127.78 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 7    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 84.206.130.249 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 8    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 193.6.241.195 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 9    0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 193.6.241.196 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 10   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 193.6.241.197 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 11   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 193.224.41.134 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 12   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 193.224.163.29 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 13   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all any all 193.225.14.141 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> 14   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all 88.218.17.33 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all any all
> 15   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all 156.230.63.70 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all any all
> 16   0     0/128  100     0        0     0    tx=0(0) rx=0(0) drp=0(0) 
> all 202.60.245.130 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all any all
> 17   0     0/128  100     0        6398  29   tx=6398(29) rx=0(0) drp=0(0)
>
> rtr2.c4e(cfg-rtr)#no flowspec-install
> rtr2.c4e(cfg-rtr)#
> rtr2.c4e(cfg)#
> rtr2.c4e#
> rtr2.c4e#show ipv4 bgp 1955 flowspec wireformat 601:1dc3:c7c7:2800::#::
> 00000000: ff ff ff ff  ff ff ff ff  ff ff ff ff  ff ff ff ff
> 00000010: 00 58 02 00  00 00 41 40  01 01 02 40  02 00 40 05
> 00000020: 04 00 00 00  64 c0 08 04  ff ff ff 01  c0 10 08 80
> 00000030: 08 07 a3 00  1e c9 82 80  09 04 c3 6f  64 a7 80 0a
> 00000040: 04 c3 6f 61  b3 80 0e 10  00 01 85 04  c3 6f 61 b3
> 00000050: 00 06 01 1d  c3 c7 c7 28
>
> rtr2.c4e#
>
>
>
>
> On 6/6/23 16:37, mc36 wrote:
> > also there are show commands to see the current dataplane policer tables 
> > like
> >
> > show ipv4 flwspc <vrf>
> >
> > after the bgp flowspec install/advertise are (re)issued, it should 
> > immediately
> >
> > show the same acl but decoded back from the bgp updates intended to the 
> > wire...
> >
> > br,
> >
> > cs
> >
> > On 6/6/23 16:34, mc36 wrote:
> > > a raw guess, have u tried cle ipv4 bgp 1 recompute instead of adding 
> > > exabgp?
> > >
> > > at the moment acls are not directly binded/linked/etc to the bgp just to 
> > > the icmp cores,
> > >
> > > so this or wait an bgp update interval is my guesss...
> > >
> > > seeking 4 your results,
> > >
> > > thx,
> > >
> > > cs
> > >
> > > On 6/6/23 16:28, David Schmitz wrote:
> > > > Hi,
> > > >
> > > > I still have some issues with the actual mitigation via Flowspec rules in 
> > > > freertr.
> > > >
> > > > Probably I am missing something.
> > > >
> > > >
> > > > To explain the issues and to allow to reproduce them,
> > > > find attached a simple test case to demonstrate and reproduce the issues.
> > > > (java -Xmx1024m -jar /rtr/rtr.jar         test tester rtr-ddos1 path ./ 
> > > > temp ./ wait)
> > > >
> > > > It consists of 3 routers: rtr1, host1, and host2.
> > > > rtr1 is supposed to perform the FlowSpec filtering of traffic between 
> > > > host1 and host2.
> > > >
> > > > rtr1#show version freeRouter-rare v23.6.6-cur, done by 
> > > > .
> > > >
> > > >
> > > > interesting part of the config for rtr1 is:
> > > >
> > > > router bgp4 1
> > > > ...
> > > >          address-family unicast flowspec
> > > >          flowspec-install
> > > >          flowspec-advert flowspec-pmap"
> > > >
> > > > policy-map flowspec-pmap
> > > >          sequence 10 action drop
> > > >          sequence 10 match access-group flowspec-rules
> > > >          !
> > > >          exit
> > > >
> > > > access-list flowspec-rules
> > > >          sequence 50 permit 1 10.1.10.1 255.255.255.255 all 10.2.10.2 
> > > > 255.255.255.255 all
> > > >          exit
> > > >
> > > > -
> > > >
> > > > The different issues I am experiencing based on this test case are 
> > > > demonstrated in 3 steps:
> > > >
> > > > step 1)
> > > >
> > > > As can be seen above,
> > > > the access-list "flowspec-rules" for policy-map "flowspec-pmap" defined 
> > > > by local config on rtr1 contains an initial rule: to block icmp traffic 
> > > > from 10.1.10.1 (host1) to 10.2.10.2 (host2).
> > > >
> > > > So, when the freertr test case rtr-ddos1.tst starts up, the filter rule
> > > > is installed as expected and actually also filterting the traffic 
> > > > correctly.
> > > > It is also nicely shown with "show policy-map flowspec":
> > > >
> > > > host2# ping 10.1.10.1 vrf CORE
> > > > ...
> > > >
> > > > rtr1# show policy-map flowspec CORE ipv4
> > > > seq         chld         queue         intrvl         byt/int         rxb 
> > > > rxp         trnsmt ace
> > > > 1                         0                                 0/128 
> > > > 100                                 0 
> > > > 192                 3                         tx=0(0) rx=0(0) drp=192(3) 
> > > > 1-1 10.1.10.1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all 10.2.10.2 
> > > > ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> > > > 2                         0                                 0/128 
> > > > 100                                 0 
> > > > 5863         118         tx=5863(118) rx=0(0) drp=0(0)
> > > >
> > > >
> > > > But, if now the filter rule is removed by changing the config, this has 
> > > > no immediate effect on the "policy-map flowspec"
> > > > and the icmp traffic from host2 to host1 continues to being filtered:
> > > >
> > > > rtr1# conf t
> > > > rtr1# access-list flowspec-rules
> > > > rtr1# no sequence 50 permit 1 10.1.10.1 255.255.255.255 all 10.2.10.2 
> > > > 255.255.255.255 all
> > > > rtr1# do wr
> > > >
> > > > rtr1# show policy-map flowspec CORE ipv4
> > > > seq         chld         queue         intrvl         byt/int         rxb 
> > > > rxp         trnsmt ace
> > > > 1                         0                                 0/128 
> > > > 100                                 0 
> > > > 448                 7                         tx=0(0) rx=0(0) drp=448(7) 
> > > > 1-1 10.1.10.1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all 10.2.10.2 
> > > > ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all
> > > > 2                         0                                 0/128 
> > > > 100                                 0 
> > > > 6357         127         tx=6357(127) rx=0(0) drp=0(0)
> > > >
> > > > Am I maybe missing something here?
> > > >
> > > >
> > > > step 2) add exabgp peering
> > > >
> > > > Now, the test case can be extended by using exabgp:
> > > >
> > > > In addition to starting the freertr test case (java -Xmx1024m -jar 
> > > > /rtr/rtr.jar test tester rtr-ddos1 path ./ temp ./ wait),
> > > > some manual additional commands are executed to start exabgp:
> > > >
> > > > # start process for tap1 interface manually         (connected to 
> > > > interface ethernet3 of rtr1)
> > > > /rtr/tapInt.bin tap1 38200 127.0.0.1 38100 127.0.0.1 # running in 
> > > > foreground
> > > >
> > > > # on another terminal:
> > > > ifconfig tap1 10.3.10.3/24
> > > >
> > > > # to install exabgp: e.g. : apt-get install exabgp
> > > > # start exabgp: will BGP connect to rtr1 (10.3.10.10 on eth3) as 
> > > > 10.3.10.3 (on tap1)
> > > > exabgp --debug ./rtr-ddos1-exabgp1.conf
> > > >
> > > > Interestingly, as soon as the peering between exabgp and rtr1 is working, 
> > > > the rule "sequence 50 permit 1 10.1.10.1 255.255.255.255 all 10.2.10.2 
> > > > 255.255.255.255 all"
> > > > already above having been deleted from the "flowspec-rules"
> > > > without formerly having an effect on "policy-map flowspec" listing,
> > > > is now immediately gone from "policy-map flowspec":
> > > >
> > > > rtr1# show policy-map flowspec CORE ipv4
> > > > seq         chld         queue         intrvl         byt/int         rxb 
> > > > rxp         trnsmt ace
> > > > 1                         0                                 0/128 
> > > > 100                                 0 
> > > > 6357         127         tx=6357(127) rx=0(0) drp=0(0)
> > > >
> > > > So, the change in the BGP connection status seems to cause an update of 
> > > > the "policy-map flowspec".
> > > >
> > > >
> > > > step 3) announce rule via exabgp
> > > >
> > > > Going on, let's dispatch a FlowSpec rule from exabgp to rtr1 via BGP:
> > > >
> > > > # example to announce a rule (same as in step 1) by local exabgp, 
> > > > exabgpcli announce flow route source-ipv4 10.1.10.1/32 destination-ipv4 
> > > > 10.2.10.2/32 protocol =icmp
> > > >
> > > > # for verification: show rules currently announced by local exabgp
> > > > exabgpcli show adj-rib out
> > > >
> > > > The new rule announced is spread to rtr1, as shown in "show policy-map 
> > > > flowspec".
> > > > But the icmp traffic from host1 (10.1.10.1) to host2 (10.2.10.2) is not 
> > > > blocked at all,
> > > > which is also reflected in the drop counter staying at zero.
> > > >
> > > >
> > > > I also tried to further extend the scenario by adding another tap2 
> > > > interface for rtr1 router and attaching an extra instance of exabgp 
> > > > peering with rtr1.
> > > > But with no success. Basically same issue, rules announced by exabgp are 
> > > > immediately listed under dynamic policy-map flowspec, but do not drop the 
> > > > traffic as expected. And unfortunately restarting the second exabgp 
> > > > instance in order to cause a bgp connection status within rtr1 do not 
> > > > help to remedy the situation unlike it helped at step 2 for locally 
> > > > emitted rules via policy-map/access-list defined by local config.
> > > >
> > > > --
> > > >
> > > > To sum it up:
> > > > - locally deleted rules (via policy-map/access-list defined in config) do 
> > > > not show up immediately after config change,
> > > > but only after a change of connection status of some BGP peering
> > > > (same seems to be happening with adding rules added via local 
> > > > policy-map/access-list)
> > > >
> > > > - FlowSpec rules added via a peering of exabgp show up immediately in the 
> > > > dynamic "policy-map flowspec", but do not actually drop the traffic as 
> > > > expected.
> > > >
> > > >
> > > > What am I missing in the BGP config or somewhere else?
> > > >
> > > >
> > > > Thanks in Advance.
> > > >
> > > >
> > > > Best Regards
> > > > David

--

David Schmitz

Boltzmannstrasse 1, 85748 Garching
Telefon:   +49 89 35831-8765
Leibniz-Rechenzentrum, Germany
Mail:  



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#1403): https://groups.io/g/freertr/message/1403
Mute This Topic: https://groups.io/mt/99364239/6413194
Group Owner: 
Unsubscribe: https://groups.io/g/freertr/unsub []
-=-=-=-=-=-=-=-=-=-=-=-

Attachment: r1-eth2--exabgppeering.pcap
Description: Binary data