RARE user and assistance email list

[mc36 <>]

hi,

i'm a bit lost now... i found two contradiction statements in your reply:

1-"I can ping between my servers with MTU 9000"
2-"switch bridge IP is limiting the MTU between the ports that are part of 
it?"

in my reading, if you can pass mtu 9000 between the servers then the switch 
is not limiting with the mtu?

thanks,
cs



On 1/6/22 16:58, Marcos Felipe Schwarz wrote:
> Thanks mc36.
>
> This seams to be dirreclty related to my issue. Somehow the cpuport is 
> limiting the MTU of the traffic.
> With the original value 8192 on /opt/freertr/bin/hwdet-main.sh, I can ping 
> the GW IP on wedge only up to 8162, which is also the value I'm limited too 
> when the problem occours (after a reboot).
> I changed the value on /opt/freertr/bin/hwdet-main.sh to 9000 and I could 
> ping the GW on wedge up to 8970, which is also the value I'm limited too when 
> the problem occours (after a reboot).
> I now set it to 9002, and now the problem has been mitigated. I can ping 
> between my servers with MTU 9000 (ping -M do 8972) after a reboot.
>
> I believe that this is not intended. Should we do any tests to verify why the 
> switch bridge IP is limiting the MTU between the ports that are part of it?
>
> -----Mensagem original-----
> De: mc36 <>
> Enviada em: quinta-feira, 6 de janeiro de 2022 10:36
> Para: ; Marcos Felipe Schwarz 
> <>
> Cc: Pedro Diniz <>; Alexander Gall <>; Jordi 
> Ortiz <>
> Assunto: Re: [RARE-users] RES: Problem on ports after reboot
>
> hi,
> let me only answer to the last, mtu part:
> so what you've measured here is the mtu that the freerouter can accomplish 
> when sending/receiving to an sdn interface...
> for that communication, the asic have the so called cpuport, and it's 
> connected to the host os as ens1... the packets are prepended a 16bit word to 
> multiplex on which source/target physical port the packet intended to/from...
> freerouter restricts itself to 1024 bytes when sending to be able to work 
> flawlessly over various tunneling topologies, setting to cpu port mtu to 8k 
> seemed more than enough... we only increased it to be able to test the jumbo 
> capabilities of the asic with locally generated packets, for the routing 
> protocols to work properly, jumbo frames are not needed at all... but yeahhh, 
> you can increase that, it's set up in /opt/freertr/bin/hwdet-main.sh on the 
> debian based images, and you should find that file if you're on a nix based 
> image....
> regards,
> cs
>
>
> On 1/6/22 13:44, Marcos Felipe Schwarz wrote:
> > Thanks for the suggestions Frederic,
> >
> > I tried some of your suggestions already. Setting autoneg to 2 doesn't work, 
> > but it does with 1. I have another server on port 19 that doesn't have this 
> > issue.
> > I'll try the other tests and post the results
> >
> > Another thing that I noticed is that when everything is worknig correctly and 
> > I can ping between Both servers with MTU 9000 and no drops, if I ping the GW 
> > IP at the Wedge I also can only get MTU 8190 to it. Is there a way to 
> > increase the MTU of a internal interface on RARE OS?
> >
> > Regards,
> >
> > Marcos Schwarz
> >
> > -----Mensagem original-----
> > De: 
> > <> Em nome de Fr  d  ric LOUI
> > Enviada em: quarta-feira, 5 de janeiro de 2022 15:16
> > Para: 
> > Cc: Pedro Diniz <>; Alexander Gall
> > <>; Jordi Ortiz <>
> > Assunto: Re: [RARE-users] Problem on ports after reboot
> >
> > Did you try to remove configuration from P4lang stanza and sdn interface and 
> > tried to add it again ?
> > Instead of plug/unplug ?
> >
> > It sounds like this is a physical BSP problem.
> >
> > Unfortunately we did not experience such problem on our case     Can you drop 
> > QSFP info ? Also this can be related to Mellanox OFED driver.
> >
> > We configured some 100GE port with CERN with Mellanox XConnect5 ofed driver 
> > and it worked flawlessly.
> >
> > Is it the only port at 100GE ? Do you have additional servers ?
> >
> > In CHICAGO we have LEONI DAC cable with 100GE port and I set AUTONEG to ON.
> > (Otherwise the link does not come up)
> > Can you please try the following line ?
> >
> > export-port sdn20 28 100 0 2 0
> >
> > The problem is that we are also using LEONI DAC but as they are using a 
> > specific P4 switch the BSP is not working correctly.
> > (No info at all from bf_platform)
> >
> > Maybe @Alex or @Jordi can share their experience as they have 100GE ports 
> > connected to Mellanox XCOnnect5 ?
> > Unfortunately in my case I usually dealt with 10GE ports.
> >
> > All in all, please to to enable autoneg and please let me know if this change 
> > something ...
>   >
> > All the best,
> > Frederic
> >
> > > Le 5 janv. 2022    16:44, Marcos Felipe Schwarz <> a   
> > > crit :
> > >
> > > Dear all,
> > >    
> > > I   m having problems on my Wedge running RARE-OS.
> > > Everytime I reboot the port sdn20 comes dropping packets and with maximum MTU 
> > > 8190 (ping -M do -s 8162). To solve the issue I need to phisically disconnect 
> > > the cable and reconnect it. If I reboot again, the problem reapear.
> > > Has any of you had a similar problem?
> > >    
> > > Port sdn20 (export-port sdn20 28 100 0 1 0) is configured with MTU 9000 and conected through a Leoni DAC cable to a Mellanox ConnectX5. I   ve tried changing the DAC cable from EdgeCore, and the problem persists.
> > > I have limited availability to the equipment, so I   d like to get some ideas 
> > > and commands to help troubleshoot the issue.
> > > This Friday I   ll be able to continue the tests, and I intend to:
> > >       Change the connection to a different port and verify if this is an port 
> > > issue
> > >       Set up logging and compare port information before and after the problem
> > >       Any other ideas that you guys can help me with to either
> > > troubleshoot or mitigate the problem
> > >    
> > > Regards,
> > >    
> > > Marcos Schwarz
> > >    
> > > Gerente de P&D | R&D Manager
> > > Ger  ncia de Execu    o de P&D em Ciberinfraestrutura | Management of R&D
> > > Execution in Cyberinfrastructure Diretoria de Pesquisa e
> > > Desenvolvimento | Board of Research and Development RNP - Rede
> > > Nacional de Ensino e Pesquisa | Brazilian National Research and
> > > Educational Network Promovendo o uso inovador de redes avan  adas |
> > > Promoting the innovative use of advanced networks http://www.rnp.br |
> > > +55 (19) 3787- 3386 | Skype ID: marcos.f.sch Campinas - SP - Brasil |
> > > E-mail: 
> > >    
> > > <RIO0001.txt>