RARE user and assistance email list

[Marcos Felipe Schwarz <>]

Thanks mc36.

This seams to be dirreclty related to my issue. Somehow the cpuport is 
limiting the MTU of the traffic.
With the original value 8192 on /opt/freertr/bin/hwdet-main.sh, I can ping 
the GW IP on wedge only up to 8162, which is also the value I'm limited too 
when the problem occours (after a reboot).
I changed the value on /opt/freertr/bin/hwdet-main.sh to 9000 and I could 
ping the GW on wedge up to 8970, which is also the value I'm limited too when 
the problem occours (after a reboot).
I now set it to 9002, and now the problem has been mitigated. I can ping 
between my servers with MTU 9000 (ping -M do 8972) after a reboot.

I believe that this is not intended. Should we do any tests to verify why the 
switch bridge IP is limiting the MTU between the ports that are part of it?

-----Mensagem original-----
De: mc36 <> 
Enviada em: quinta-feira, 6 de janeiro de 2022 10:36
Para: ; Marcos Felipe Schwarz 
<>
Cc: Pedro Diniz <>; Alexander Gall <>; 
Jordi Ortiz <>
Assunto: Re: [RARE-users] RES: Problem on ports after reboot

hi,
let me only answer to the last, mtu part:
so what you've measured here is the mtu that the freerouter can accomplish 
when sending/receiving to an sdn interface...
for that communication, the asic have the so called cpuport, and it's 
connected to the host os as ens1... the packets are prepended a 16bit word to 
multiplex on which source/target physical port the packet intended to/from...
freerouter restricts itself to 1024 bytes when sending to be able to work 
flawlessly over various tunneling topologies, setting to cpu port mtu to 8k 
seemed more than enough... we only increased it to be able to test the jumbo 
capabilities of the asic with locally generated packets, for the routing 
protocols to work properly, jumbo frames are not needed at all... but yeahhh, 
you can increase that, it's set up in /opt/freertr/bin/hwdet-main.sh on the 
debian based images, and you should find that file if you're on a nix based 
image....
regards,
cs


On 1/6/22 13:44, Marcos Felipe Schwarz wrote:
> Thanks for the suggestions Frederic,
> 
> I tried some of your suggestions already. Setting autoneg to 2 doesn't 
> work, but it does with 1. I have another server on port 19 that doesn't 
> have this issue.
> I'll try the other tests and post the results
> 
> Another thing that I noticed is that when everything is worknig correctly 
> and I can ping between Both servers with MTU 9000 and no drops, if I ping 
> the GW IP at the Wedge I also can only get MTU 8190 to it. Is there a way 
> to increase the MTU of a internal interface on RARE OS?
> 
> Regards,
> 
> Marcos Schwarz
> 
> -----Mensagem original-----
> De:  
> <> Em nome de Fr  d  ric LOUI 
> Enviada em: quarta-feira, 5 de janeiro de 2022 15:16
> Para: 
> Cc: Pedro Diniz <>; Alexander Gall 
> <>; Jordi Ortiz <>
> Assunto: Re: [RARE-users] Problem on ports after reboot
> 
> Did you try to remove configuration from P4lang stanza and sdn interface 
> and tried to add it again ?
> Instead of plug/unplug ?
> 
> It sounds like this is a physical BSP problem.
> 
> Unfortunately we did not experience such problem on our case     Can you 
> drop QSFP info ? Also this can be related to Mellanox OFED driver.
> 
> We configured some 100GE port with CERN with Mellanox XConnect5 ofed driver 
> and it worked flawlessly.
> 
> Is it the only port at 100GE ? Do you have additional servers ?
> 
> In CHICAGO we have LEONI DAC cable with 100GE port and I set AUTONEG to ON.
> (Otherwise the link does not come up)
> Can you please try the following line ?
> 
> export-port sdn20 28 100 0 2 0
> 
> The problem is that we are also using LEONI DAC but as they are using a 
> specific P4 switch the BSP is not working correctly.
> (No info at all from bf_platform)
> 
> Maybe @Alex or @Jordi can share their experience as they have 100GE ports 
> connected to Mellanox XCOnnect5 ?
> Unfortunately in my case I usually dealt with 10GE ports.
> 
> All in all, please to to enable autoneg and please let me know if this 
> change something ...
 > 
> All the best,
> Frederic
> 
>> Le 5 janv. 2022    16:44, Marcos Felipe Schwarz <> a  
>>  crit :
>>
>> Dear all,
>>   
>> I   m having problems on my Wedge running RARE-OS.
>> Everytime I reboot the port sdn20 comes dropping packets and with maximum 
>> MTU 8190 (ping -M do -s 8162). To solve the issue I need to phisically 
>> disconnect the cable and reconnect it. If I reboot again, the problem 
>> reapear.
>> Has any of you had a similar problem?
>>   
>> Port sdn20 (export-port sdn20 28 100 0 1 0) is configured with MTU 9000 
>> and conected through a Leoni DAC cable to a Mellanox ConnectX5. I   ve 
>> tried changing the DAC cable from EdgeCore, and the problem persists.
>> I have limited availability to the equipment, so I   d like to get some 
>> ideas and commands to help troubleshoot the issue.
>> This Friday I   ll be able to continue the tests, and I intend to:
>>      Change the connection to a different port and verify if this is an 
>> port issue
>>      Set up logging and compare port information before and after the 
>> problem
>>      Any other ideas that you guys can help me with to either 
>> troubleshoot or mitigate the problem
>>   
>> Regards,
>>   
>> Marcos Schwarz
>>   
>> Gerente de P&D | R&D Manager
>> Ger  ncia de Execu    o de P&D em Ciberinfraestrutura | Management of R&D
>> Execution in Cyberinfrastructure Diretoria de Pesquisa e 
>> Desenvolvimento | Board of Research and Development RNP - Rede 
>> Nacional de Ensino e Pesquisa | Brazilian National Research and 
>> Educational Network Promovendo o uso inovador de redes avan  adas | 
>> Promoting the innovative use of advanced networks http://www.rnp.br |
>> +55 (19) 3787- 3386 | Skype ID: marcos.f.sch Campinas - SP - Brasil |
>> E-mail: 
>>   
>> <RIO0001.txt>
>