Rare project developers

[Alexander Gall <>]

Here is a (not so) brief summary for this list of what I took out of a
VC I've had with csaba and Gawen regarding the "bspless deployment". I
hope it contains some useful information for the folks here :)

First, to clarify what exactly we're talking about, because there was
a misunderstanding among us when we talked about "QSFP modules working
or not".  For csaba it means that he can bring a port up and send and
receive packets. For Gawen and me it means that as well (of course)
but also that the software can access the plugin through its I2C
management interface.

For SFP/QSFP and friends (but not, e.g. QSFP-DD) it is true that at
least for some types of plugins, all you need is the physical
connection of the serdes lanes to bring up a link. You won't be able
to interact with the plugin in any way, e.g. to determine it's
properties, monitor operational parameters or, in the case of QSFP,
disable and enable lasers (among other things). Even just resetting
the module is not possible since that requires driving a separate pin
which is also platform dependent (usually through a GPIO controller).

Providing access to the QSFP plugins (this is meant to include SFPs
via an adapter) via I2C is a large part of what a BSP is used for. Due
to the nature of I2C, this can't be handled by generic code.

The Intel SDE uses an API to encapsulate the platform-specific stuff
in a few shared libraries. Every BSP produces an instance of these
libraries for a specific platform. In a regular BSP-based
installation, bf_switchd uses those libraries to interact with the
platform, e.g. the QSFP modules but also power supplies and thermal
sensors.

The BSP also provides another element that's specific to a platform
called the "port map", which contains very low-level information about
the physical wiring of the Tofino serdes ports, e.g. which lane of a
Tofino MAC block is connected to which lane of a QSFP cage. This
information is programmed into the ASIC as part of the initialization
performed by bf_switchd (maybe similar to how auto-MDI/MDIX works, not
sure). At least that's how I understand it, please correct me if I'm
wrong :)

Enter "bspless mode". The only thing that's absolutely necessary for
bf_switchd to operate is to have the correct port map to match a given
platform. Usually, this is provided by one of the platform-specific
libraries from the platform's BSP.  If we build the SDE completely
without BSP, those libraries are missing and so is the information
about the port map.  In this case, bf_switchd assumes that we want to
operate on the Tofino model and automatically switches to that mode,
i.e. the P4 program will not be loaded on the ASIC.

However, there is also a way to supply the port map as a JSON file
through one of the build artifacts of the P4 program that we want to
run. Csaba has explained how to do this earlier in this thread and
also in the Intel forum
(https://community.intel.com/t5/Intel-Connectivity-Research/Installing-SDE-without-corresponding-BSP/m-p/1365676#M3253).
If bf_switchd can't find the platform libraries but it gets the port
map via the program's configuration, it does not switch to the model
but proceeds to load and run the program on the ASIC.

This is what we call "bspless mode". Due to the missing libraries,
bf_switchd cannot access the QSFP modules in any way. In the UCLI
(bf-shell), the entire bf_pltfm context is missing, including
everything concerning other parts of the chassis (power supplies, fans
etc.).

Csaba's suggestion is that we could support platforms for which we
don't have a BSP in this bspless mode (provided that we can get our
hands on the port map files), since even that very limited support is
better than no support at all. I tend to agree that this could cover
some cases where that's good enough, e.g. equipment in a lab or some
other place that's close to the operator using non-fancy plugins like
DAC cables.

This should actually not be difficult to support with the RARE
packaging, so I think I'll give that a try :)

However, I believe that this particular road ends with Tofino1. I'm
pretty sure that bspless mode is simply impossible for platforms that
use QSFP-DD, i.e. starting with Tofino2.  These types of plugins
require interaction with the host to operate properly. I know for a
fact that that's the case for the SDE when running on Tofino2.

I also believe that we should *not* go bspless for platforms that have
a BSP but don't support the newest SDE that we would otherwise like to
use (e.g. right now we're limited to 9.7.2 in our current RARE release
due to this problem). bspless should only be a last resort for
otherwise unsupported platforms.

I'd welcome feedback of any kind :)

Thanks for listening.

-- 
Alex


On Mon, 4 Jul 2022 12:31:41 +0200, mc36 <> said:

> well, not....
> all you need is
> - astock debian stable
> - the sde
> - and the patch you did against the sde
> - the yaml you gave me to compile for asic but without bsp
> - the portmap from stratum
> - mentioning port in the bf-p4c's output bf_router.json

> br,
> cs





> On 7/4/22 10:49, Gawen Davey wrote:
>> This requires ONLPv2.
>> So ONL, but I do have a build guide for ONLP on Ubuntu. But it may be for 
>> ONLPv1. As far as I remember, ONLPv2 was essentially abandoned -- There 
>> were two commits this year, then 
>> it goes back to 2020. And we certainly haven't maintained anything ONL for 
>> a long time.
>> 
>> If you want to give it a go anyway: 
>> https://github.com/APS-Networks/ubuntu-onlp 
>> <https://github.com/APS-Networks/ubuntu-onlp>
>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> *From:* mc36 <>
>> *Sent:* 04 July 2022 08:44
>> *To:*  <>; Alexander Gall 
>> <>
>> *Cc:* Gawen Davey <>; Fr  d  ric LOUI 
>> <>; Alexander Jeffries 
>> <>
>> *Subject:* Re: [rare-dev] Access
>> look at this:
>> 
>> https://github.com/stratum/stratum/blob/753270b6061c5efe15ee60178169b8abe032f65e/stratum/hal/bin/barefoot/deb/start-stratum.sh#L54-L63
>>  
>> <https://github.com/stratum/stratum/blob/753270b6061c5efe15ee60178169b8abe032f65e/stratum/hal/bin/barefoot/deb/start-stratum.sh#L54-L63>
>> especially the #L61, that is, everything except the wedges, they bring 
>> them up without a bsp!!!
>> qsfps work fine without bsp... serdes too... okk, sensors are really 
>> missing but heyy...
>> 
>> also: 
>> https://github.com/stratum/stratum/blob/main/stratum/hal/bin/barefoot/README.build.md
>>  
>> <https://github.com/stratum/stratum/blob/main/stratum/hal/bin/barefoot/README.build.md>
>>   
>> // ctrl+f: recommended mode
>> 
>> gawen, how you see it?
>> 
>> thanks,
>> cs
>> 
>> 
>> 
>> On 7/4/22 09:20, Alexander Gall wrote:
>>> On Fri, 1 Jul 2022 19:10:27 +0200, mc36 <> said:
>>> 
>>>> what about this one? :)
>>>> https://github.com/stratum/stratum/blob/main/stratum/hal/config/x86-64-inventec-d10064-r0/port_map.json
>>>>  
>> <https://github.com/stratum/stratum/blob/main/stratum/hal/config/x86-64-inventec-d10064-r0/port_map.json>
>>> 
>>> Same thing. This will set up the serdes to match the board, but
>>> without the BSP none of the i2c-related stuff works. Actually, the
>>> Inventec is even more complicated because some of the
>>> platform-dependent stuff is currently still supplied by the ONL layer
>>> (i2c enumeration, HW sensors, QSFP eeprom access etc.), not the
>>> BSP. And the ONL stuff doesn't work on Debian 11 out of the box :(
>>>