RARE user and assistance email list

["mc36" <>]

documenting....


-------- Forwarded Message --------
Subject: Re: freeRouter rpms for CentOS and Fedora
Date: Tue, 1 Feb 2022 07:01:29 +0100
From: mc36 <>
Reply-To: 
To: 

> For example if I have 10 cores 0-9 and add kernel parameter isolcpus=1-9
> reserving cores 1-9 for dpdk and leaving core 0 for freerouter process,
> how to best allocate 9 cores for two dpdk interfaces?

with that particular one, i would give the following additional parameters
(without considering the cpu->socket and nic->socket mapping...)

0 1 2  \  ! do port0 rx on core1 and port0 tx on core2
1 3 4  \  ! do port1 rx on core3 and port1 tx on core4
-1 5 6 \  ! core 5 and 6 only do packet processing
-1 7 7 \  ! core 7 also do only packet processing

an alternative, if you see that packet io does not consume too much cpu,
you can allocate more for packet processing and less for packet io:

0 1 1  \  ! do port0 rx/tx on core1
1 2 2  \  ! do port1 rx/tx on core2
-1 3 4 \  ! core 3 and 4 only do packet processing
-1 5 6 \  ! core 5 and 6 also do only packet processing
-1 7 7 \  ! core 7 also do only packet processing

or even you can completely collapse the packet io to a single core:

0 1 1  \  ! do port0 rx/tx on core1
1 1 1  \  ! do port1 rx/tx on core1
-1 2 3 \  ! core 3 and 4 only do packet processing
-1 4 5 \  ! core 5 and 6 also do only packet processing
-1 6 7 \  ! core 6 and 7 also do only packet processing

and finally a very simple example without dedicated packet processing cores:

0 1 2  \  ! do port0 rx&processing on core1 and port0 tx on core2
1 3 4  \  ! do port1 rx&processing on core3 and port1 tx on core4

regards,
cs




> Thanks,
> Alexey
>
> 2022-01-31 17:39 GMT+02:00, mc36 <>:
> > hi,
> >
> > On 1/31/22 16:22,  wrote:
> > > Hi Csaba,
> > >
> > > > so --dev is a dpdk config option... is the dpdk compiled with the
> > > > af-packet
> > > > pmd?
> > >
> > > In CentOS 8 repository dpdk compiled without af-packet.
> > > I compiled it in my repository with af-packet and it starts with
> > > --vdev=net_af_packet0,iface=veth250.
> >
> > well done!
> >
> > > Packets now arriving but I found problems with jumbo frames.
> > >
> > > pmtud-reply does not reply as with p4emu,
> > >
> > > reply on ping to IP address of freertr interface with mtu 9000
> > > received only for size 1938 "ping 10.2.2.1 -s 1938" if size between
> > > 1939 and 1972
> > > than reply packet have the same size 1980 as -s 1938 and bad checksum,
> > > if -s larger 1972 then no reply (sdn2 rx increases, tx don't,
> > > ethernet0 rx/tx no changes)
> > >
> > > interface  state  tx             rx             drop
> > > ethernet0  up     479255         457376          0
> > > sdn2       up     316830+572198  293291+834006  0+0
> > >
> > > I applied what wiki say
> > > ethtool --offload veth250 rx off tx off sg off tso off ufo off gso off
> > > gro off lro off rxvlan off txvlan off rxhash off
> > > ethtool --offload veth251 rx off tx off sg off tso off ufo off gso off
> > > gro off lro off rxvlan off txvlan off rxhash off
> > > but nothing changed.
> >
> > this is a limitation how p4dpdk picks up and sends the packets:
> > so the thing is that this 1920 is the dpdk's default buffer size
> > minus the dpdk's default headroom... at the moment i have no idea
> > on how to tweak that programmatically...
> >
> >
> > > > you can remap your ports to different cores to free up the console of
> > > > p4dpdk.bin
> > > > by adding 0 1 1   1 1 1  2 1 1  3 1 1 to your cli... then you can press
> > > > "i" to see the
> > > > packet counters... do they increase? do you see packets back and forth
> > > > on
> > >
> > > What do these numbers mean and where need to add (example how to do it
> > > if possible)?
> >
> > so two types of such extra parameters exists for the p4dpdk code:
> > the one is <port> <rxcore> <txcore> and the other is -1 <forwardercore>
> > <forwardercore>
> > the first maps the receive and transmit processing of a given port...
> > if the second form is not used, then the packet processed on the receiver
> > core...
> > if any of the second format appears, then the receiver and transmitter does
> > only packet-io, and the mentioned forwarder cores will do the processing...
> > this latter lets you efficiently use a machine with a lot of available
> > cores,
> > with small but very fast interfaces... then for every packet received, a
> > quick
> > hash will be computed, and the packet will be processed on the hashed
> > core...
> >
> > regards,
> > cs
> >
> >
> >
> >
> > > Thanks,
> > > Alexey
> > >
> > > 2022-01-31 8:14 GMT+02:00, mc36 <>:
> > > > hi,
> > > > i go inline...
> > > > regards,
> > > > cs
> > > >
> > > >
> > > > On 1/31/22 04:03,  wrote:
> > > > > Hi Csaba,
> > > > >
> > > > > 2022-01-22 8:24 GMT+02:00, mc36 <>:
> > > > > > here i would suggest to continue experimenting this with the qcow2,
> > > > > > and follow the 'activate exetrnal forwarding' lines from freertr.net.
> > > > > > it'll enable you the pcap based dataplane, you'll see how it works,
> > > > > > then you'll be able to replace the p4emu with p4dpdk in your centos
> > > > > > installation...
> > > > >
> > > > > I made working configuration with p4emu
> > > > niceee.....
> > > >
> > > > > but when trying to replace it
> > > > > with p4dpdk it doesn't work.
> > > > >
> > > > > With p4emu I run
> > > > > /usr/bin/p4emu.bin 127.0.0.1 9080 3 eth1 eth2 eth3 veth250
> > > > >
> > > > > with p4dpdk
> > > > >
> > > > > /usr/bin/dpdk-devbind.py -b uio_pci_generic 0000:00:05.0
> > > > > /usr/bin/dpdk-devbind.py -b uio_pci_generic 0000:00:06.0
> > > > > /usr/bin/dpdk-devbind.py -b uio_pci_generic 0000:00:07.0
> > > > >
> > > > > next command I found here
> > > > > https://wiki.geant.org/pages/viewpage.action?pageId=148084865
> > > > > but it failed
> > > > >
> > > > > /bin/p4dpdk.bin --vdev=net_af_packet0,iface=veth250 127.0.0.1 9080 3
> > > > > EAL: failed to parse device "net_af_packet0"
> > > >
> > > > so --dev is a dpdk config option... is the dpdk compiled with the
> > > > af-packet
> > > > pmd?
> > > >
> > > >
> > > > > It starts with net_tap0 but no traffic on freertr.
> > > > in thetory, net-tap also could work for the cpu-port....
> > > >
> > > > > How p4dpdk should be started?
> > > > you're doing it right, that should work...
> > > > things to check for, if you see the packets arriving? you can remap your
> > > > ports to
> > > > different cores to free up the console of p4dpdk.bin by adding 0 1 1   1
> > > > 1 1
> > > >    2 1 1  3 1 1
> > > > to your cli... then you can press "i" to see the packet counters... do
> > > > they
> > > > increase?
> > > > do you see packets back and forth on port 3 (your selected cpu port) ?
> > > > do you see them arriving at freertr's eth0 interface?
> > > > if you cannot see any packets at all in dpdk from the nics, maybe you
> > > > should
> > > > give a try
> > > > binding to a different vfio driver.... in kvm vms, i see
> > > > uio_pci_generic,
> > > > but for real
> > > > hardware without virtualization, i see vfio-pci to work...
> > > >
> > > >
> > > >
> > > > > Thanks,
> > > > > Alexey


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#134): https://groups.io/g/freertr/message/134
Mute This Topic: https://groups.io/mt/88966636/6413194
Group Owner: 
Unsubscribe: https://groups.io/g/freertr/unsub []
-=-=-=-=-=-=-=-=-=-=-=-