RARE user and assistance email list

[Gabriel Tetzner <>]

Hi Frédéric,

Thanks for your answer, I think looking for the solution by seeing the topology and imagining the structure worked.

I'll update the github with new documentation.

Att: Gabriel Tetzner

Em qua., 7 de set. de 2022 às 11:04, Frédéric LOUI <> escreveu:

Hi Gabriel,

I have to congratulate you !

You totally nailed it !

To be honest, my perception of your problem was a guesstimate :)

For -1-, you found the solution I wanted you to guess ! :)

And in order to accomplish this you have several solution:
   ->  I would suggest you use pcapInt with a pair of veth
   -> With rawInt, let’s say … you’d just have raw packet
   -> tap is also indeed a valid solution
   -> ...

-2- yes, if you need your container to be aware of the host networking context this « host » mode is needed
     But … please keep in mind that if you are using DPDK (Unix networking context is useless …)
     So in that case you’d need to start P4DPDK and add this unix interface during freeRtr/P4DPDK CLI launch

-3- I feel you. It is very satisfying to see all these graphs. Please use latest dashboard, but also look at updated doc. (Prometheus agent has to be altered)
     And you’ll be rewarded with Graph having hostname and not their «target instance value » :)
     If you also could update RARE/freeRtr/Prometheus doc, that would be super cool   (https://github.com/rare-freertr/freeRtr-prometheus)

Sorry for not having more explicit, but I felt that guiding you toward the right direction was more fruitful for you than giving you the configs/answers without explanation.
In the long term you would be able to develop a strong and efficient analysis WRT network traffic flows. (And this will be useful for you in the future if you would like to start Container Container Networking with K8s for example)

Cheers & congrats again !

All the best,
Frederic

> Le 7 sept. 2022 à 15:32, Gabriel Tetzner <> a écrit :
>
> Hi all,
>
> Frédéric, I have good news,
>
> Apparently everything worked out fine...
>
> I did a nice exercise of reviewing and understanding what was behind the topology, continuing with what was already done, basically I took three points into consideration:
>
> 1 - I needed a bridge between my freeRtr with the debian host machine, for this I created a tap interface for this, I take into consideration that the tap of the freeRtr is a standard one for telnet access and I didn't want to mess up the experiments, this tip is needed for the vm to reach the other end of the tunnel, as the tap would be kind of a gateway for it to pass through the freeRtr (I don't know if it was clear).
>
> 2 - As you say you need the docker host to be in the same network as the other freeRtr and it recognize inside the tunnel network, I think this was easy to solve with Csaba's help, with the suggestion to create the container and then install freeRtr to map this new network, but you can do it later with rawInt.bin
>
> 3 - Finally, I made sure to put the prometheus-agent in the new vrf I created to isolate it and made a server forwarder to access prometheus, grafana, and the prometheus-agent.
>
> Now I have a monitoring running on top of an IP tunnel from another network provider, I found this magnificent, it is basically a monitoring aiming to reach an isolated tunnel from another network, I still believe I have to adjust some things and do some proof testing still.
>
> Please, I just have to thank you for your help with the suggestions and the ways you gave, I believe this was a basic problem, but for me it was a complex network and monitoring challenge.
>
> <image.png>
>
> Note: This is still the old dashboard.
>
> Prometheus:
> <image.png>
> Thank you.
>
> Att: Gabriel Tetzner
>
> Em qua., 31 de ago. de 2022 às 14:05, Frédéric LOUI <> escreveu:
> HI Gabriel,
>
> Don’t over complexity things. In order to better breakdown your problem step by step.
>
> You can deal with your problem without freeRtr first in order to better understand the network plumbing.
>
> Replace all yours host by Debian/Ubuntu VM:
>
> - interconnect them with your favorite Virtual machine bridge or point to point link
> - Create Wireguard between them with ubuntu (you’ll find a lot of resource on the web)
> - Enable Linux forwarding
> - Start with with Linux static routes
>
> Check linux routing with: ip route show
>
> Once you get your overlay up and running (underlay being basic linux interconnect) you can fire up your prometheus conrtainer with docker daemon
>
> Try to establish the connectivity you’d like to achieve.
> When you’ll solve this and understood how and why connectivity works you’d be able to figure out what you need when you’ll use freeRtr
>
> This is actually not a freeRtr problem but mostly a very good  network routing exercise.
>
> Good luck,
> Frederic
>
> > Le 31 août 2022 à 17:49, Gabriel Tetzner <> a écrit :
> >
> > Hi Frédéric,
> >
> > Good,
> > I'll try to do that, there was an advice from Csaba to install freeRouter after I run the container and docker installation, I think it's fair to try this configuration...
> > I'll give you more details later, here in this thread.
> >
> > Thank you,
> >
> > Att: Gabriel Tetzner
> >
> > Em qua., 31 de ago. de 2022 às 11:24, Frédéric LOUI <> escreveu:
> > Hi Gabriel,
> >
> > I think that I almost start to figure out what you are trying to do … But still don’t have a clear idea.
> > Generally speaking forget now the underlay as you mention that your overlay is working and that you are able to ping.
> >
> > So now please draw a L3 diagram with your overlay only and indicate IP accordingly on each segment.
> >
> > Once you got your diagram, now think about how your prometheus server is connected to freertr-rtr3 _AT_LAYER_3.
> > I’m not sure if your prometheus container is running in host mode or is NAT’ed behind docker0 but if you think network plumbing but you need to figure out how the prometheus server is discussing with the outer world.
> >
> > Next question you need to figure out is: how freertr-r3 is seeing this docker bridge.
> >
> > show ipv4 route <YOUR_OVERLAY_VRF>
> >
> > Should give you the answer. If you don’t see docker network subnet, it means that freertr-r3 does not know how to reach prometheus server
> >
> > In order to simplify the problem I would use an external prometheus/grafana server and make sure that the prometheus subnet is seen and  advertised in your network.
> >
> > Once it is working, just replicate the setup but with docker network environment.
> >
> > Your real problem here are not the Wireguard tunnels here. Your solution is to figure out how to make freertr-r3 aware of docker subnet.
> > Once this subnet is « Injected » into the VRF you’d just have to advertise it with any routing protocol of your choice.
> >
> > Maybe you’ll have to use pcapInt and a combination of veth pair and configure it into the hw config file
> >
> > Sorry but again as it is unclear from my part this is just a guess ...
> >
> > > Le 31 août 2022 à 15:46, Gabriel Tetzner <> a écrit :
> > >
> > > Hi Frederic,
> > >
> > > answering the first question, I want a tunnel wireguard from another network to reach the end of another tunnel wireguard from another network.
> > >
> > > A little crazy I know haha,
> > >
> > > answering the question, with Csaba's help I chose a routing protocol and isolated the underlay and overlay with VRF, and I am using the EIGRP4 protocol for routing, I wanted to know if this is a good protocol as well, some attempts gave error, but it worked in the end.
> > >
> > > "What is the IP of the Debian machine ? Is it the machine running freeRtr ? A machine connected to freertr-r3 ?  " ->
> > >
> > > So it would be the machine running freeRouter, this machine is in the cloud, however I have a prometheus server on it, and it is not the prometheus agent but the server running in docker on the machine with freeRouter:
> > >
> > > https://prometheus.io/
> > >
> > > <image.png>
> > >
> > > This machine in the topology would be the freeRtr-r3, it is in aws and its IP configuration:
> > >
> > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN group default qlen 1000
> > >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > >     inet 127.0.0.1/8 scope host lo
> > >        valid_lft forever preferred_lft forever
> > >     inet6 ::1/128 scope host
> > >        valid_lft forever preferred_lft forever
> > > 2: enX0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
> > >     link/ether 02:b1:02:dc:c6:18 brd ff:ff:ff:ff:ff:ff
> > > 3: br-122d8c7c1e50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
> > >     link/ether 02:42:61:21:c7:60 brd ff:ff:ff:ff:ff:ff
> > >     inet 172.18.0.1/16 brd 172.18.255.255 scope global br-122d8c7c1e50
> > >        valid_lft forever preferred_lft forever
> > > 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
> > >     link/ether 02:42:f3:e6:60:b3 brd ff:ff:ff:ff:ff:ff
> > >     inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
> > >        valid_lft forever preferred_lft forever
> > > 10: tap20001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
> > >     link/ether 00:00:4e:2c:4e:2b brd ff:ff:ff:ff:ff:ff
> > >     inet 10.255.255.1/24 scope global tap20001
> > >        valid_lft forever preferred_lft forever
> > >     inet6 2001:db8:ffff:ffff:200:4eff:fe2c:4e2b/64 scope global dynamic mngtmpaddr
> > >        valid_lft 2419189sec preferred_lft 604789sec
> > >     inet6 fe80::200:4eff:fe2c:4e2b/64 scope link
> > >        valid_lft forever preferred_lft forever
> > > 23381: veth4b2630d@if23380: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-122d8c7c1e50 state UP group default
> > >     link/ether d2:6f:3c:b2:e3:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
> > > 23383: vethdad3c9c@if23382: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-122d8c7c1e50 state UP group default
> > >     link/ether 5e:d6:da:31:4a:2f brd ff:ff:ff:ff:ff:ff link-netnsid 1
> > >
> > > "In this topology consider that there is a prometheus server running on top of freeRtr-r3 debian and I need to talk to 10.0.0.3? »
> > > Do you mean freertr server running Prometheus agent ?
> > >
> > > Answer: As I mentioned in the text, the prometheus agent will only be on the other end (freeeRtr-r2), the prometheus server is running in a container on the debian machine freeRtr-3.
> > >
> > > Now the big question arises for me:
> > >
> > > I need the debian machine where freeRtr-3 is running, to reach the ip of the other end, in my view if freeRouter reaches through a tunnel wireguard and a routing protocol, the host machine (debian) of freeRtr should recognize it too.
> > >
> > > Or the way I set it up is also wrong, but I think there is a way, either create a tap, or connect another tunnel wireguard from inside freeRtr-r3 to inside her host vm, so there will be communication.
> > >
> > > I need this so that the prometheus server can reach the endpoint of the tunnel wireguard of freeRtr-r2 so that it can monitor the network interfaces of freeRtr-r2 through an agent running on top of it.
> > >
> > > Here is a picture of the topology to help:
> > >
> > > <image.png>
> > >
> > >
> > > Background: I am working on this topology to ensure that there is monitoring with network isolation.
> > >
> > > Honestly I am lost, I need help...
> > >
> > > Thank you,
> > >
> > > Att: Gabriel Tetzner
> > >
> > >
> > >
> > >
> > >
> > > Em qua., 31 de ago. de 2022 às 06:27, Frédéric LOUI <> escreveu:
> > > Hi Gabriel,
> > >
> > > Good to see that you did progress thanks to Csaba’s advice.
> > > Looking at your question, something is not clear for me, what are you trying to achieve ?
> > > Which end to end connectivity do you want to enable here?
> > >
> > > You mention: "debian machine in freeRtr-r3 ping the ip of the freeRtr-2 tunnel"
> > >
> > > What is the IP of the Debian machine ? Is it the machine running freeRtr ? A machine connected to freertr-r3 ?
> > >
> > > You also mention:
> > > "In this topology consider that there is a prometheus server running on top of freeRtr-r3 debian and I need to talk to 10.0.0.3? »
> > > Do you mean freertr server running Prometheus agent ?
> > >
> > > Is it possible to share configs of r1/r2 and r3 ?
> > >
> > > Or please elaborate.
> > >
> > > Thanks
> > > Frederic
> > >
> > > > Le 26 août 2022 à 22:17, Gabriel Tetzner <> a écrit :
> > > >
> > > > Now I am left with a question, how do I make the debian machine in freeRtr-r3 ping the ip of the freeRtr-2 tunnel?
> > > >
> > > > In this topology consider that there is a prometheus server running on top of freeRtr-r3 debian and I need to talk to 10.0.0.3?
> > >
> >
>