RARE user and assistance email list

[Frédéric LOUI <>]

Hi Marc,

Sorry for the late reply. 
The RARE project created a ONIE image to remove the hassle to install 
everything by hand.

If you want to run RARE on the Wedge100BF32X, please use this documentation 
page:
https://docs.rare.geant.org/guides/installation/wedge100bf32/onie-nos-install/

If you follow this page instruction, all should be installed for you. 

From there you should be able to be able to upgrade to latest version.
If you have some issue, which I hope not :-) we can have a quick zoom session 
so that you can replay this installation.

From that point you should be able to configure the switch using:
https://docs.rare.geant.org

Basically Ctrl+F in reference page in order to find a feature.

All the best,
Frederic


> Le 6 sept. 2023 à 10:19, Alexander Gall <> a écrit :
> 
> 
> BTW, you can use editcap to drop the custom 2-byte header if you want
> to see the original packets with tcpdump
> 
> $ editcap -C 2 /tmp/control_plane_traffic.pcap - | tcpdump -n -r -
> reading from file -, link-type EN10MB (Ethernet), snapshot length 262144
> 10:13:36.551297 LLDP, length 118: tofino0
> 10:13:36.627536 LLDP, length 118: tofino0
> 10:14:06.550730 LLDP, length 118: tofino0
> 10:14:06.627968 LLDP, length 118: tofino0
> 10:14:09.765610 ARP, Request who-has 10.0.1.1 tell 10.0.1.1, length 34
> 10:14:09.767709 ARP, Request who-has 10.0.2.1 tell 10.0.2.1, length 34
> 10:14:12.813289 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
> 10:14:12.813378 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
> 10:14:13.814050 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
> 10:14:13.814131 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
> 10:14:14.814777 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
> 10:14:14.814898 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
> 10:14:15.815549 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
> 10:14:15.815661 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
> 10:14:16.816327 ARP, Request who-has 10.0.2.202 tell 10.0.2.1, length 34
> 10:14:16.816429 ARP, Reply 10.0.2.202 is-at e4:1d:2d:f2:96:78, length 48
> 
> --
> Alex
> 
> On Wed, 6 Sep 2023 09:43:08 +0200, Alexander Gall <> said:
> 
>> Hi
> 
>> On Wed, 6 Sep 2023 08:32:05 +0900, Marc Bruyere 
>> <> said:
> 
>>> Sorry the bf-shell was from one of our UFI came from another platoform 
>>> where we have the similar issues.
> 
>> Ok, that was really confusing :)
> 
>>> But here we try to at least make works the WEDGE reference platform first.
> 
>>> Changed the page with the correct output from the WEDGE
>>> https://md.bruyere.family/ImBJBzbJSkaYoVmez9ZfFg?view
> 
>>> But better to have a look and do pause on the screen recording
>>> https://cloud.bruyere.family/s/2oebQ9RgkJW5PeE
> 
>>> We do not have issue with the kernel PCI module and we did loaded 
>>> withbf_kpkt_mod_load.
>>> The issue is the packet are corrupted as seen in the pcap trace attached 
>>> in the previous email.
> 
>> I haven't looked at the recording yet, but note that the packets on
>> the control-plane interface use a non-standard header. The first two
>> bytes is the physical port ID from where the packet originated or
>> where it needs to be sent (it is set or read by the P4 program). For
>> example
> 
>>        0x0000:  0098 ffff ffff ffff 80a2 35f0 82b4 0806
>>        0x0010:  0001 0800 0604 0001 80a2 35f0 82b4 0a00
>>        0x0020:  0201 0000 0000 0000 0a00 02ca 0000 0000
> 
>> is an ARP packet (Ethertype 0x0806) coming from port 0x0098
>> (152). It's a request from 80:a2:35:f0:82:b4 (10.0.2.1) to resolve
>> 10.0.2.202.
> 
>> This looks ok. There is actually a response
> 
>>        0x0000:  0000 80a2 35f0 82b4 e41d 2df2 9678 0806
>>        0x0010:  0001 0800 0604 0002 e41d 2df2 9678 0a00
>>        0x0020:  02ca 80a2 35f0 82b4 0a00 0201 0000 0000
>>        0x0030:  0000 0000 0000 0000 0000 0000 0000
> 
>> The port ID here is 0. I'd have expected 0x0098 here as well. Freertr
>> can't associate this reply with the correct interface, I guess.
> 
>> I haven't seen this before, not sure what to make of it. The official
>> RARE build uses SDE 9.11.2, not sure if that could be a problem.
> 
>> BTW, is there a particualr reason why you don't use our ONIE installer
>> to set up your system? It would help to eliminate differences in the
>> build and the environment as the source of this problem.
> 
>> -- 
>> Alex
> 
>>> Any idea, why ?
> 
>>> /mb
> 
> 
> 
> 
>>> Le 2023/09/05 à 22:00, Alexander Gall a écrit :
>>>> Hi
>>>> 
>>>> On Tue, 5 Sep 2023 17:38:46 +0900, Marc Bruyere 
>>>> <> said:
>>>> 
>>>>> Hi All,
>>>>> We are having issues to make it work RARE/FreeRtR on our WEDGE Tofino1.
>>>> I'm a bit confused: here you say your device is a WEDGE, i.e. EdgeCore
>>>> DCS800 (formerly known a Wedge100bf-32x), but at the link below you
>>>> say it's a "UFI s9180-32x". I assume it's the latter :)  Since you
>>>> seem to build all software from scratch, you would need a "BSP"
>>>> (Baseboard Support Package) supplied by the vendor to get full support
>>>> of the hardware (I didn't know about this platform until now so we
>>>> also don't support it in our regular RARE builds). It should still
>>>> work without it but you won't be able to see the QSFP plugins from
>>>> within bfshell.
>>>> 
>>>> However, your first problem seems fairly simple. From your bfshell
>>>> log:
>>>> 
>>>> ASIC detected at PCI /sys/class/bf/bf0/device
>>>> ERROR getting ASIC pci device id fpr /sys/class/bf/bf0/device
>>>> Starting PD-API RPC server on port 9090
>>>> bf_switchd: drivers initialized
>>>> error opening /dev/bf0
>>>> ERROR: Device mmap failed for dev_id 0
>>>> Please load driver with bf_kdrv_mod_load script. Exiting..
>>>> 
>>>> This means that the kernel module hasn't been loaded yet. For RARE
>>>> you'll need the bf_kpkt module. So please use the bf_kpkt_mod_load
>>>> script from the SDE to load the module. This will create a new network
>>>> interface called "bf_pci0" (might get renamed to something else by
>>>> udev), which is needed for the the control-plane traffic between the
>>>> Tofino and freertr.
>>>> 
>>>> I hope this will bring you a step closer to success.
>>>> 
>>>> --
>>>> Alex
>>>> 
>>>>> We have been able to run control plane and date plane, but we see like 
>>>>> corrupted packets reaching the FreeRouter process.
>>>>> Please also find a short screen recording : 
>>>>> https://cloud.bruyere.family/s/2oebQ9RgkJW5PeE
>>>>> We tried to ping from lab 2 to the freerouter interface but freerouter 
>>>>> does not even reply to ARP request has its not recognize the packets.
>>>>> We tried to ping from freerouter to lab 2 same issue.
>>>>> and a pcap trace of the traffic capture on the ubuntu PCIE ens1 
>>>>> interface attached to this email.
>>>>> More details on this page 
>>>>> https://md.bruyere.family/ImBJBzbJSkaYoVmez9ZfFg?view
>>>>> and our testbed diagram :
>>>>> +----+---------------------------+---+
>>>>> |    +---------------------------+   |
>>>>> |    |  FreeRouter               |   |
>>>>> |    |                           |   |
>>>>> |    |                           |   |
>>>>> |    |                           |   |
>>>>> |    |                           |   |
>>>>> |    |  10.0.1.1       10.0.2.1  |   |
>>>>> |    |  sdn3              sdn4   |   |
>>>>> |    +--+-------------------+----+   |
>>>>> |       |                   |        |
>>>>> |       |                   |        |
>>>>> |   FrontPanel 3        FrontPanel4  |
>>>>> |   Internal 144        Internal 152 |
>>>>> +-------+-------------------+--------+
>>>>> |                   |
>>>>> |                   |
>>>>> |                   |
>>>>> |                   |
>>>>> |                   |
>>>>> |                   |
>>>>> |                   |
>>>>> |                   |
>>>>> 10.0.1.101    |                   |     10.0.2.202
>>>>> +-------+----+         +----+--------+
>>>>> | Lab1 server|         | Lab2 Server |
>>>>> |            |         |             |
>>>>> |            |         |             |
>>>>> |            |         |             |
>>>>> |            |         |             |
>>>>> |            |         |             |
>>>>> |            |         |             |
>>>>> +------------+         +-------------+
>>>>> Your help will be very appreciated !
>>>>> Marc and Benoit
>>>>> /mb