RARE user and assistance email list

[Alexander Gall <>]

BTW, you can use editcap to drop the custom 2-byte header if you want
to see the original packets with tcpdump

$ editcap -C 2 /tmp/control_plane_traffic.pcap - | tcpdump -n -r -
reading from file -, link-type EN10MB (Ethernet), snapshot length 262144
10:13:36.551297 LLDP, length 118: tofino0
10:13:36.627536 LLDP, length 118: tofino0
10:14:06.550730 LLDP, length 118: tofino0
10:14:06.627968 LLDP, length 118: tofino0
10:14:09.765610 ARP, Request who-has 10.0.1.1 tell 10.0.1.1, length 34
10:14:09.767709 ARP, Request who-has 10.0.2.1 tell 10.0.2.1, length 34
10:14:12.813289 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
10:14:12.813378 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
10:14:13.814050 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
10:14:13.814131 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
10:14:14.814777 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
10:14:14.814898 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
10:14:15.815549 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
10:14:15.815661 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
10:14:16.816327 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
10:14:16.816429 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48

--
Alex

On Wed, 6 Sep 2023 09:43:08 +0200, Alexander Gall <> said:

> Hi

> On Wed, 6 Sep 2023 08:32:05 +0900, Marc Bruyere <> 
> said:

>> Sorry the bf-shell was from one of our UFI came from another platoform 
>> where we have the similar issues.

> Ok, that was really confusing :)

>> But here we try to at least make works the WEDGE reference platform first.

>> Changed the page with the correct output from the WEDGE
>> https://md.bruyere.family/ImBJBzbJSkaYoVmez9ZfFg?view

>> But better to have a look and do pause on the screen recording
>> https://cloud.bruyere.family/s/2oebQ9RgkJW5PeE

>> We do not have issue with the kernel PCI module and we did loaded 
>> withbf_kpkt_mod_load.
>> The issue is the packet are corrupted as seen in the pcap trace attached 
>> in the previous email.

> I haven't looked at the recording yet, but note that the packets on
> the control-plane interface use a non-standard header. The first two
> bytes is the physical port ID from where the packet originated or
> where it needs to be sent (it is set or read by the P4 program). For
> example

>         0x0000:  0098 ffff ffff ffff 80a2 35f0 82b4 0806
>         0x0010:  0001 0800 0604 0001 80a2 35f0 82b4 0a00
>         0x0020:  0201 0000 0000 0000 0a00 02ca 0000 0000

> is an ARP packet (Ethertype 0x0806) coming from port 0x0098
> (152). It's a request from 80:a2:35:f0:82:b4 (10.0.2.1) to resolve
> 10.0.2.202.

> This looks ok. There is actually a response

>         0x0000:  0000 80a2 35f0 82b4 e41d 2df2 9678 0806
>         0x0010:  0001 0800 0604 0002 e41d 2df2 9678 0a00
>         0x0020:  02ca 80a2 35f0 82b4 0a00 0201 0000 0000
>         0x0030:  0000 0000 0000 0000 0000 0000 0000

> The port ID here is 0. I'd have expected 0x0098 here as well. Freertr
> can't associate this reply with the correct interface, I guess.

> I haven't seen this before, not sure what to make of it. The official
> RARE build uses SDE 9.11.2, not sure if that could be a problem.

> BTW, is there a particualr reason why you don't use our ONIE installer
> to set up your system? It would help to eliminate differences in the
> build and the environment as the source of this problem.

> -- 
> Alex

>> Any idea, why ?

>> /mb




>> Le 2023/09/05 à 22:00, Alexander Gall a écrit :
>>> Hi
>>> 
>>> On Tue, 5 Sep 2023 17:38:46 +0900, Marc Bruyere 
>>> <> said:
>>> 
>>>> Hi All,
>>>> We are having issues to make it work RARE/FreeRtR on our WEDGE Tofino1.
>>> I'm a bit confused: here you say your device is a WEDGE, i.e. EdgeCore
>>> DCS800 (formerly known a Wedge100bf-32x), but at the link below you
>>> say it's a "UFI s9180-32x". I assume it's the latter :)  Since you
>>> seem to build all software from scratch, you would need a "BSP"
>>> (Baseboard Support Package) supplied by the vendor to get full support
>>> of the hardware (I didn't know about this platform until now so we
>>> also don't support it in our regular RARE builds). It should still
>>> work without it but you won't be able to see the QSFP plugins from
>>> within bfshell.
>>> 
>>> However, your first problem seems fairly simple. From your bfshell
>>> log:
>>> 
>>> ASIC detected at PCI /sys/class/bf/bf0/device
>>> ERROR getting ASIC pci device id fpr /sys/class/bf/bf0/device
>>> Starting PD-API RPC server on port 9090
>>> bf_switchd: drivers initialized
>>> error opening /dev/bf0
>>> ERROR: Device mmap failed for dev_id 0
>>> Please load driver with bf_kdrv_mod_load script. Exiting..
>>> 
>>> This means that the kernel module hasn't been loaded yet. For RARE
>>> you'll need the bf_kpkt module. So please use the bf_kpkt_mod_load
>>> script from the SDE to load the module. This will create a new network
>>> interface called "bf_pci0" (might get renamed to something else by
>>> udev), which is needed for the the control-plane traffic between the
>>> Tofino and freertr.
>>> 
>>> I hope this will bring you a step closer to success.
>>> 
>>> --
>>> Alex
>>> 
>>>> We have been able to run control plane and date plane, but we see like 
>>>> corrupted packets reaching the FreeRouter process.
>>>> Please also find a short screen recording : 
>>>> https://cloud.bruyere.family/s/2oebQ9RgkJW5PeE
>>>> We tried to ping from lab 2 to the freerouter interface but freerouter 
>>>> does not even reply to ARP request has its not recognize the packets.
>>>> We tried to ping from freerouter to lab 2 same issue.
>>>> and a pcap trace of the traffic capture on the ubuntu PCIE ens1 
>>>> interface attached to this email.
>>>> More details on this page 
>>>> https://md.bruyere.family/ImBJBzbJSkaYoVmez9ZfFg?view
>>>> and our testbed diagram :
>>>> +----+---------------------------+---+
>>>> |    +---------------------------+   |
>>>> |    |  FreeRouter               |   |
>>>> |    |                           |   |
>>>> |    |                           |   |
>>>> |    |                           |   |
>>>> |    |                           |   |
>>>> |    |  10.0.1.1       10.0.2.1  |   |
>>>> |    |  sdn3              sdn4   |   |
>>>> |    +--+-------------------+----+   |
>>>> |       |                   |        |
>>>> |       |                   |        |
>>>> |   FrontPanel 3        FrontPanel4  |
>>>> |   Internal 144        Internal 152 |
>>>> +-------+-------------------+--------+
>>>> |                   |
>>>> |                   |
>>>> |                   |
>>>> |                   |
>>>> |                   |
>>>> |                   |
>>>> |                   |
>>>> |                   |
>>>> 10.0.1.101    |                   |     10.0.2.202
>>>> +-------+----+         +----+--------+
>>>> | Lab1 server|         | Lab2 Server |
>>>> |            |         |             |
>>>> |            |         |             |
>>>> |            |         |             |
>>>> |            |         |             |
>>>> |            |         |             |
>>>> |            |         |             |
>>>> +------------+         +-------------+
>>>> Your help will be very appreciated !
>>>> Marc and Benoit
>>>> /mb