FELIX_DEVICEROUTESOURCEADDRESS limiatation in multi node cluster

[elszmad]

I have a 3 node kubernetes cluster (k3s 1.30, but it does not matter).
I'm using POD CIDR range 192.168.0.0/16
Calico create a routing entry for every POD ip like this:
192.168.23.63 dev caliXXXXX src uid 0

When I create a new routing table in /etc/iproute2/rt_tables, define a default gw in this table and an ip rule for source-based routing for ip address of my first interface it will break the calico routing entries.
I found this FELIX_DEVICEROUTESOURCEADDRESS parameter what seems to solve my issue, but in a 3 node setup I should use 3 different sources addresses, but this env enable to define only 1 address. For my problem it would be fine to be able to set an interface instead of an ip address.

Expected Behavior

Calico routing entries should work with source-based routing.

Current Behavior

If we define source-based routing with default gw i the first interface it will break calico routing and FELIX_DEVICEROUTESOURCEADDRESS won't solve this problem in multi node kubernetes setup.

Possible Solution

Introdice a FELIX_DEVICEROUTESOURCEINTERFACE when an interface name can be specified.

Steps to Reproduce (for bugs)

server1:
ip addr add 10.0.0.1/24 dev eth0
echo "200 mytable" >> /etc/iproute2/rt_tables
ip route add default via 10.0.0.254 table mytable
ip rule add from 10.0.0.2 table mytable
ip addr add 10.0.1.1/24 dev eth1
ip route add default via 10.0.1.254

server2:
ip addr add 10.0.0.2/24 dev eth0
echo "200 mytable" >> /etc/iproute2/rt_tables
ip route add default via 10.0.0.254 table mytable
ip rule add from 10.0.0.2 table mytable
ip addr add 10.0.1.2/24 dev eth1
ip route add default via 10.0.1.254

server3:
ip addr add 10.0.0.3/24 dev eth0
echo "200 mytable" >> /etc/iproute2/rt_tables
ip route add default via 10.0.0.254 table mytable
ip rule add from 10.0.0.3 table mytable
ip addr add 10.0.1.3/24 dev eth1
ip route add default via 10.0.1.254

Install kubernetes to these servers, install calico

Context

Your Environment

Calico version 3.28.0
k3s 1.30.0
SLES 15SP5

[caseydavenport]

I found this FELIX_DEVICEROUTESOURCEADDRESS parameter what seems to solve my issue, but in a 3 node setup I should use 3 different sources addresses,

To set a different value for each node, you would need to create three per-host FelixConfiguration objects.

https://docs.tigera.io/calico/latest/reference/resources/felixconfig

The resources with the name node. contain the node-specific overrides, and will be applied to the node .

[caseydavenport]

When I create a new routing table in /etc/iproute2/rt_tables, define a default gw in this table and an ip rule for source-based routing for ip address of my first interface it will break the calico routing entries.

Could I ask what it is you're trying to achieve with this setup?

[mazdakn]

Also be aware the Calico manages a range of route table which is configurable by routeTableRanges in FelixConfiguration object as mentioned above. By default the range value is 1 to 250. If you want to add a route table you should not pick a number from this range.
https://docs.tigera.io/calico/latest/reference/resources/felixconfig

[elszmad]

When I create a new routing table in /etc/iproute2/rt_tables, define a default gw in this table and an ip rule for source-based routing for ip address of my first interface it will break the calico routing entries.

Could I ask what it is you're trying to achieve with this setup?

I have separate networks in different interfaces, if a request arrive to 10.0.0.1 (eth0) from a different network we have to route the response back from this source ip (if I have only 1 default gw in the main table it will route everything in 1 interface)

[elszmad]

I found this FELIX_DEVICEROUTESOURCEADDRESS parameter what seems to solve my issue, but in a 3 node setup I should use 3 different sources addresses,

To set a different value for each node, you would need to create three per-host FelixConfiguration objects.

https://docs.tigera.io/calico/latest/reference/resources/felixconfig

The resources with the name node. contain the node-specific overrides, and will be applied to the node .

Can you give me a simple example, from the https://docs.tigera.io/calico/latest/reference/resources/felixconfig it does not seem trivial for me.

Thanks,
Laszlo

[caseydavenport]

If you want to configure a specific node, you can create a FelixConfiguration:

kind: FelixConfiguration
apiVersion: projectcalico.org/v3
metadata:
  name: node.mynodename-01
spec: {}

[elszmad]

Thanks for the support! This solves my problem.