Podman doesn't set up firewall for use for 22.03.5

[davidtluong]

Maintainer: @oskarirauta
Environment: (put here arch, model, OpenWrt version)
ARMv8 Processor rev 3
Raspberry Pi Compute Module 4 Rev 1.0
22.03.5

Description:
Podman port mapping via exposing the port via -p flag for run doesn't work. It opens the port on the server, however, it doesn't properly forward the traffic to the container. I did notice I still had to install iptables, despite the switch from that this version. Installed crun, cni-plugins, and podman initially. Then I had ran into the following, my rc.local runs 2 containers, 1 forwards 7406 to 8080 on the container, works fine for 21.x.x, the following is for the reported version:

sh /etc/rc.local 
WARN[0000] Failed to load cached network config: network podman not found in CNI cache, falling back to loading network podman from disk 
WARN[0000] 1 error occurred:
	* plugin type="bridge" failed (delete): cni plugin bridge failed: failed to locate iptables: exec: "iptables": executable file not found in $PATH
 
Error: plugin type="bridge" failed (add): cni plugin bridge failed: failed to locate iptables: exec: "iptables": executable file not found in $PATH

opkg install iptables

h /etc/rc.local 
WARN[0000] Failed to load cached network config: network podman not found in CNI cache, falling back to loading network podman from disk 
Error: plugin type="bridge" failed (add): cni plugin bridge failed: running [/usr/sbin/iptables -t nat -C CNI-9d7039292401a53dae6302d5 ! -d 224.0.0.0/4 -j MASQUERADE -m comment --comment name: "podman" id: "3a86e6897c3eac52a6866fee61ebdefaf2ef5522cd89ae1fc9eb673bb3fc2179" --wait]: exit status 2: iptables v1.8.7 (nf_tables): Chain 'MASQUERADE' does not exist
Try `iptables -h' or 'iptables --help' for more information.

opkg install kmod-ip-nat

sh /etc/rc.local 
632a79862aeb46a4fc5b622b4761600df990a1b75f2725975e1062b36ab3b380
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 82e490cc2043 done  
Copying blob 598a42ec6587 done  
Copying blob e4cad15ac3f6 done  
Copying blob e886f0f47ef5 done  
Copying blob 9a9138853e32 done  
Copying blob 948128637a91 done  
Copying blob 096332b242c2 done  
Copying config 2a4fbb36e9 done  
Writing manifest to image destination
Storing signatures
c823412a9f2c08ac8848c6f9d2fc4acf36141c6d733f90d36b42f1bbac333dda

curl -v http://10.88.0.5:8080
> GET / HTTP/1.1
> Host: 10.88.0.5:8080
> User-Agent: curl/8.3.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Length: 0
< 

curl -v http://10.0.0.1:7406
> GET / HTTP/1.1
> Host: 10.0.0.1:7406
> User-Agent: curl/8.3.0
> Accept: */*
> 

Hitting the pod IP from the server is successful (I expect the not found since I didn't hit the path), hitting the server IP port does not, it just stalls.

[oskarirauta]

It shouldn't work. I think I posted guide somewhere on how to use it. On openwrt, you don't use firewalling by podman, instead you set it up and do your own firewall rules. Every time your firewall refreshes, your podman generated rules are flushed. No point when you already are using a dist that has one of its main focus on firewalling. It's not a bug, it's a feature ;)

If you want to use firewalling by podman, use it on something that utilises systemd..

Also, if I remember correctly, by using Podman's own fw management, you end up in world of pain. It triggered firewall renewal and removes route from lan to podman network. If you still insist on doing it the podman way - you can setup that route back by yourself. Though, it's flushed every time firewall renews. Which can happen for numerous reasons (vpn, etc. can trigger it). I had it once like that. Wouldn't recommend it.

Anyway, check next 2 posts. The first one should suite your setup the best. Let me know how it went.

[oskarirauta]

@rh-dluong

Found my old setup, should work perfectly for you.

/etc/cni/net.d/87-podman-bridge.conflist:

{
  "cniVersion": "0.4.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [{ "dst": "0.0.0.0/0" }],
        "ranges": [
          [
            {
              "subnet": "10.129.0.0/24",
              "gateway": "10.129.0.1"
            }
          ]
        ]
      }
    },
    {
      "type": "portmap",
      "snat": false,
      "capabilities": {
        "portMappings": false
      }
    },
    {
      "type": "firewall",
      "backend": "iptables"
    },
    {
      "type": "tuning"
    }
  ]
}

/etc/config/network:

config interface 'podman'
	option proto 'none'
	option device 'cni-podman0'

/etc/config/firewall:

config zone
	option name		podman
	list   device		'cni-podman0'
	list   subnet		'10.129.0.0/24'
	option input		REJECT
	option output		ACCEPT
	option forward		ACCEPT
	option masq		0
	option mtu_fix		1

config forwarding
	option src		lan
	option dest		podman

config forwarding
	option src		wan
	option dest		podman

config forwarding
	option src		podman
	option dest		wan

config redirect
	option name		Allow-HTTP
	option src		wan
	option dest		podman
	option src_dport	80
	option dest_ip		10.129.0.2
	option dest_port	80
	option proto		tcp
	option target		DNAT

config redirect
	option name		Allow-HTTPS
	option src		wan
	option dest		podman
	option src_dport	443
	option dest_ip		10.129.0.2
	option dest_port	443
	option proto		tcp
	option target		DNAT

This setup ensures that lan can connect to containers, but not the other way around. Containers have access to wan and if port forwarding are set up (check allow-http and allow-https), wan also has access to containers.

Just forget about that podman based firewalling, for containers running in a long-term, it's dead idea any way in a setup like this. Fits more if you are using a desktop computer that uses something to open ports, for duration of software running. This also gives you full control over firewall rules, if you want to setup something special.

[oskarirauta]

@rh-dluong

This is for the new way with netavark and cni-protocol package installed.

/etc/containers/networks/podman.json:

{
     "name": "podman",
     "id": "5ef894788befd4d42498314b6e66282ca730aa2e1e82f9b9597bf4d1725ca074",
     "driver": "bridge",
     "network_interface": "podman0",
     "created": "2023-02-20T08:56:34.652030952Z",
     "subnets": [
          {
               "subnet": "10.129.0.0/24",
               "gateway": "10.129.0.1"
          }
     ],
     "ipv6_enabled": false,
     "internal": false,
     "dns_enabled": true,
     "ipam_options": {
          "driver": "host-local"
     }
}

/etc/config/network:

config interface 'podman0'
	option proto 'cni'
	option device 'podman0'

/etc/config/firewall:

config zone
        option name 'podman'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option mtu_fix '1'
        list network 'podman'
        list network 'podman0'

config forwarding
        option src 'lan'
        option dest 'podman'

config forwarding
        option src 'podman'
        option dest 'wan'

config forwarding
        option src 'wan'
        option dest 'podman'

config redirect
        option name 'Allow-HTTP'
        option src 'wan'
        option dest 'podman'
        option src_dport '80'
        option dest_ip '10.129.0.102'
        option dest_port '80'
        option proto 'tcp'
        option target 'DNAT'
        option reflection '0'

[oskarirauta]

Oh, and one more thing. Forget about dhcp; if you going to run anything that you want to expose to the world- statically set ip address for container/pod is your choice.

If this solves your problem(s), would you mind changing title of the issue to something wider? Just so anyone having network configuration issues with podman, could find it easier in the future?

[davidtluong]

Hmmm... Unfortunately the 1st post didn't work out for me, getting the same behaviour still. I'll wait for the update so I can implement the 2nd post. What version is that slated for? I can change the title regardless, thanks for the detailed info!

[oskarirauta]

@rh-dluong

current podman package version is 4.6.2-2.

you'd need to update to git version of openwrt for that.. Though you seem to have decent enough openwrt, if you check the errors; you see that it's missing iptables, if you don't disable firewall/portmapping - you need to install compatibility package, since it's missing because you have nftables, not iptables.. podman/docker/etc don't yet support nftables out of the box.. To avoid error message, you also could setup a dummy iptables command.

just a script that outputs fake iptables version number, such as:
/usr/bin/iptables:

#!/bin/sh
echo "iptables v1.8.7 (legacy)"

Also try adding this to your rc.local after creating your containers:

LAN_IPADDR=$(uci get network.lan.ipaddr)
ip route del 10.129.0.0/24
ip route add 10.129.0.0/24 dev cni-podman0 proto kernel scope link via 10.129.0.1 src ${LAN_IPADDR}

or at least try to use this to setup route from terminal first to see if it works..

That will create route. I had it on my old setup (the first post). Ofcourse, if you have different subnet for podman, adjust it. This must be executed every time you create a new container. My current/newer setup does not have/need this.

[oskarirauta]

Any success? I am just updating my host and can give you a working example in few days, if you didn't yet figure it out; although, it's with newest components, such as netavark and podman 4.7.0. But might still be very helpful.

[davidtluong]

Unfortunately not, I think I'm just gonna have to wait until the appropriate podman version is released where I can use the podman.json method of doing it.

[oskarirauta]

podman.json? That's something new. Out of curiosity, where could I find information about that?
If it is something like kube configs, it's not going to fix the problem. But I can assure you that podman does work, though I cannot promise that version that you are running would be something that I could help with at the moment, but I'll try..

Which version of podman are you running at the moment, and what version of OpenWrt, if git/snapshot/nightly, which revision or approx date of build?

I just got my host on the other side of the world to work with brand new openwrt build + podman, but can't yet publish details of how to get it to work here, because I need to do some more tests to see if I can reduce necessary steps.. When I'm finished, I'll share my setup here, though it's quite similar to setup 1 that I shared earlier. Currently I am hosting a proxy, nginx and php in single pod/own containers with success. To get it to work properly was a bit difficult as I ran to problems as well, dns was not shared, and such..

[davidtluong]

That's what you had for /etc/containers/networks/podman.json in #22255 (comment)

The openwrt version:
22.03.5

The podman version:
podman-4.0.2

I had to revert back to 21.02.7 and podman-3.4.4 to get it in a working state.

[oskarirauta]

Oh, that's netavark's config.. It's been there for sometime, but as it's written in rust....
I recall that there was issues along the way, got those fixed somewhere around version 4.3..
It just might be quite some time until new major version of openwrt pops out; that's pretty much when it comes available unless you go with nightlies..

[davidtluong]

I'll just have to wait around I suppose, I rely on these containers for my homelab and wireguard. I really do appreciate all your efforts on this!

[oskarirauta]

File transfer (about 2.5gb) finished and I rebooted. Works as it should. Containers cannot reach lan, but they can be reached from lan. Proxy serves website from nginx as it should and firewall makes the magic happen. I didn't even need that route fixing. I can recommend it.

/etc/containers/networks/podman.json:

{
     "name": "podman",
     "id": "5ef894788befd4d42498314b6e66282ca730aa2e1e82f9b9597bf4d1725ca074",
     "driver": "bridge",
     "network_interface": "podman0",
     "created": "2023-02-20T08:56:34.652030952Z",
     "subnets": [
          {
               "subnet": "10.129.0.0/24",
               "gateway": "10.129.0.1"
          }
     ],
     "ipv6_enabled": false,
     "internal": false,
     "dns_enabled": true,
     "ipam_options": {
          "driver": "host-local"
     }
}

/etc/config/network:

config interface 'podman0'
	option proto 'cni'
	option device 'podman0'
	option target '10.129.0.0/24'
	option gateway '10.129.0.1'
	option source '10.100.0.1'
	option disabled '1'

/etc/hotplug.d/10-cni

#!/bin/sh
[ $(uci get network.${INTERFACE}.proto) = "cni" ] && {
	[ "$ACTION" = add ] && ifup ${INTERFACE}
	[ "$ACTION" = remove ] && ifdown ${INTERFACE}
	[ "$ACTION" = change ] && {
		ifdown ${INTERFACE}
		sleep 1
		ifup ${INTERFACE}
	}
}

/etc/config/firewall:

config zone
	option name		podman
	list network		'podman0'
	option input		REJECT
	option output		ACCEPT
	option forward		ACCEPT
	option mtu_fix		1

config forwarding
	option src		lan
	option dest		podman

config forwarding
	option src		podman
	option dest		wan

config forwarding
	option src		wan
	option dest		podman

config rule
	option name 'Allow-Podman-DNS'
	option src 'podman'
	list dest_ip '10.129.0.1'
	option dest_port '53'
	option target 'ACCEPT'

config redirect
	option name		Allow-HTTP
	option src		wan
	option dest		podman
	option src_dport	80
	option dest_ip		10.129.0.2
	option dest_port	80
	option proto		tcp
	option reflection	0
	option target		DNAT
	option enabled		1

config redirect
	option name		Allow-HTTPS
	option src		wan
	option dest		podman
	option src_dport	443
	option dest_ip		10.129.0.2
	option dest_port	443
	option proto		tcp
	option reflection	0
	option target		DNAT
	option enabled		1

pod + containers are created from scripts called by rc.local on boot, here's pod creation:

#!/bin/sh

podman pod create \
		--replace \
		--name servers \
		--hostname srv \
		--ip 10.129.0.2

podman pod start servers

Then I just add containers to pod. I have scripts for those too, but I am quite sure that you know enough about that. In the pod creation, important thing was that ip designation. How otherwise could I configure my firewall to redirect to correct ip address, if it wouldn't be static....

principle:

• Proxy answers at :80 and :443
• nginx is on :8080 and php-fpm on :9000
• ports :80 and :443 are redirected
• there's a redirection from :80 to :443 to force https/ssl.
• allow connection to 10.129.0.1 but only to port 53 (dns)

Next part is unnecessary after adding firewall rule to allow access to port 53.

Special thing that I had to do, was that wan wasn't that accessible from containers as it was supposed to; or it was, but dns wasn't working. My containers are based on alpine, so dns is configured in /etc/resolv.conf, so I add this to my container creation:
--volume /srv/shared/resolv.conf:/etc/resolv.conf:Z,rw.

And my /srv/shared/resolv.conf looks like this:

search dns.podman
nameserver 10.129.0.1
nameserver 1.1.1.1

Probably it would be possible to work this out, for example with enabling dnsmasq on podman0 even, hard to say; this works and I'm good with it.

Caveats:
If you try to access your outbound ip address (internet ip address, wan address) from container - connection does not fails. I haven't so far figured out a firewall rule to work this out. Even any from podman zone to directly to ip address through any: accept, fails. This though rarely is needed, if you run a server on your pod, you could as well connect to 127.0.0.1 or container's ip address......

EDIT: Improved configuration with hotplug script, more settings for interface and some improves to firewall rules.

This is with today's git openwrt + packages where podman version is 4.7.0 and netavark at 1.8.0.
03.10.2023

   ░░░░░░  ░░░░░░░ ░░░░░░░ ░░░   ░░ ░░░  ░░░░░░░░░░  ░░░░░░░░░
  ░▒█████░ ░██▓███░░▓█████░███▄  ██░██░ ░░░█▓▒█████▄░▒█████▓▒░
 ░▒██▒  ██▒▒██▒ ▀██▒▓█▒░░▀ ██▀█░ ██▒▓█░░█▒░█▓▒▓█▒ ▀██░ ▒██▒░░░
 ░▒██░ ░██▒▒██░  █▓▒████░  ██░▀█ ██▒░█▒░█▒░█▒░▓█░ ▄█▀  ▒██▒░  
 ░░██░ ░██░▒██▄▄█▓░▒▓█▒░░▄ ██▒ █▌██▒░█▒▒██░█▒▒██▀██▄   ░█▓▒░  
  ░▒████▓▒░▒██▒░░░░▒██████▒██░ ▀███░ ░██▒██░░░██ ░▒██░ ░██▒░  
   ░░░▒▒░░ ░▓█▒░   ░░░░░▒░░▒░░  ░░▒░ ░░░░▒░  ░░▒  ░▒░░ ░░▒░░  
                            W I R E L E S S    F R E E D O M  
 ----------------------------------------------------------------
 OpenWrt SNAPSHOT, git commit r24069-954142f477
 ----------------------------------------------------------------
root@kvm:~#

[davidtluong]

This'll be good reference when netavark and cni-protocol releases in OpenWRT, right now, I'm not advanced enough with OpenWRT to start using the snapshots.

[oskarirauta]

Apologies 😔

…

This'll be good reference when netavark and cni-protocol releases in OpenWRT, right now, I'm not advanced enough with OpenWRT to start using the snapshots.

[oskarirauta]

I've set up now backport updates to all relevant packages about podman for openwrt-23.05, so I would recommend you to update to 23.05, it takes some time until packages are available, but when I checked, all packages, including netavark and support for it, is already available on 23.05. Just bit out-dated.

If pull requests are merged, in quite short time you should have available all the tools available in git/unstable version of openwrt as up-to-date versions.

Pr's made are:
#22394 #22395 #22396 #22397 #22398 #22399 #22400

Only relevant update that I left out for now is catatonit, which has PR on the master branch on-going, but that doesn't matter, as catatonit 0.1.7 - 0.2.0 is technically the same, they just changed their licensing.
Changelog

And why I didn't make PR's for 22.03.05 is because I couldn't include netavark (and aardvark-dns, which in this case is optional component) as a package, since it's written in rust, which isn't supported in older than 23.05. I would though recommend you to update to 23.05 anyway.

[oskarirauta]

All merged. Available after next rebuild of openwrt-23.05 package repository (probably happens weekly or something like that)

[davidtluong]

No need to apologize, you've definitely went above and beyond here. I have a few presentations I need to finish off and I'll take a look into this!

[syb999]

port via

Please try adding "--network host" parameter because I used this parameter in version 23.05.0 to resolve the access issue when running the Docker container for clouddrive2.
Now port mapping can be directly configured in the OpenWrt firewall.
Note, make sure that iptables-nft is installed.

[davidtluong]

So, I tried this, and unfortunately the port forwarding situation isn't going to work as it is in the example. I have want my nginx server exposed via lan, not wan, and I can't port forward all 80/443 traffic from lan to podman as this breaks everyrthing else. The old expose method worked as it mapped the external port for the lan IP to podman's container as expected.

[oskarirauta]

@rh-dluong

Yes, that is a known problem. You might succeed though if you disable your luci or configure it to not occupy ports 80/443 in all interfaces like it does as default. Depending on configuration, that will be all that is needed (it's a long time when I conducted these tests). This way you will be able to connect to it at least from host running podman, surely with proper network setup, from lan as well.

Only problem that I haven't been able to find a way around, is to connect wan ip from lan, so that it would redirect to container ip, from lan I can only connect to container with container's ip address, from wan, redirection works; as long as I am outside of lan.. otherwise I am forced to use container's ip directly. I am pretty sure that all other scenarios have been tested by me quite profoundly.

[davidtluong]

I have already configured luci for different ports as I have it running this way in version 21.x.x. I have allowed lan zone to podman zone forwarding, it opens the port on the lan IP, but it doesn't actually connect to the container itself. It will work if I port forward, but as I can't redirect all 80/443 traffic as that'll break external traffic, that isn't a solution. I can connect to the Podman IP from the host, but I cannot connect to it outside the host. I use this nginx server as a load balancer for my Openshift (kubernetes) cluster apps and the api will connect correctly to the 3 master nodes.

[oskarirauta]

@rh-dluong

I actually haven't tried that and you might be on to something.
Sounds like a routing issue. Check routes on computer's in your lan and try to set a route manually. Quite complicated setup for openwrt, but I see no reason why it wouldn't work out; just needs some work. Also, try to ping container ip from lan, to see if it's completely un-routable, or just dropping connections. That's a some kind of starting point..

[davidtluong]

Hmmm... not sure what the difference is, but port mapping works just fine in fedora. I'm unfortunately using docker for now since portmapping is working as expected in OpenWRT. Not my preference tho.

[Wabuo]

Hi @oskarirauta

First, thanks for the awesome tutorials and the pioneering.

I'm trying to set up containers providing internal services only, and it's doing my head in.

What am I trying to achieve:
Grafana instance; LAN only
Blocky or AdGuard home; LAN and internet access for the container.

Maybe a future wiki and chat server that would be reachable from the internet and from local LAN.

As in, I basically want the opposite to your Nginx example, but cant seem to get it to work, do you have an example network for that too? I guess it would be nice to have both working at the same time, with the ability for containers to join both networks.

Is there a way to just use the br-lan interface and have containers appear with a mac address like a random physical device and let OpenWrt deal with the rest?

[Wabuo]

Oh also Netavark is currently working on native nftables support see this draft-pr containers/netavark#883

I guess if we want to influence that in any way, to make sure it plays well with running on a device without systemd and frequently reloads the firewall, now is a good time.

[Wabuo]

Im also trying to work on dockerman support for podman, but don't expect too much, it's my first dabble in to Lua and the Luci world.

The issue is here, lisaac/luci-app-dockerman#181. No pull request yet, still reading up on luci-app development, not sure if it ever even will develop to that stage.

[oskarirauta]

@wabue

Most containers available always need tweaking and I always do my own. Openwrt also isn't designed explicitly for podman, and podman isn't doing a lot to support for openwrt- and it's fine. Still their fusion works very well.

But blocky/adguard combination.. I would forget that in the first place. Use native adguard-home if you need it. Grafana also sounds like something that is designed to be running on a different host but with work, at least parts of it might be possible; though I would forget about k8s part from the start. For me, I see what you are trying to do there; I have something similar on my network, but not with openwrt, you should use something else for that. It probably not suites your needs, but for example, TrueNAS Scale(still considering that Core would be more my choice) that I run on my file server works better for your plans. Just like so many other systems.

Re-check my guide here then re-check your firewall and network configuration. If you find configuring openwrt firewall difficult or insist on --expose being the way you want to, please, run podman on something else than openwrt... Most proposals you have seem to me like a risk of complete network.

My nginx configuration is running on my web server flawlessly, has been running there for years, current uptime for that server is 47 days. It is also behind a proxy that handless my ssl and also uses php container. I have given my network configurations in my guide. They are pretty much identical actually. I hope the issue isn't with containers mentioned as those urls are examples that probably do not even exist.

There is luckily no way to just use br-lan with podman. Not with openwrt, or any other system and thanks to god there isn't. Running servers that way would be completely insane! And I am aware of nftables support in netavark, it is already integrated to my last podman/netavark packages, check here, line 77.
Wrapper script is no longer necessary for netavark. Just uncomment line 77.

What comes to exposing ports with podman, I stand behind my recommendation to forget it as on openwrt hosting podman, you really should instead handle it by openwrt; if you find it difficult to manage network/firewall on openwrt after my guide, considernce again not to run podman on openwrt; it's not designed to be running on a openwrt, instead all the bells and whistles are available with something that runs systemd.

With my instructions you can run it on openwrt but you must realise risks and do a proper configurations to make it safe enough, in my opinion, whole expose command is just a back door to your systems if firewall isn't configured well enough. Openwrt has a very good firewall and also incredibly good web interface to manage it, and for --expose on openwrt, that is an easy way towards a disaster. I don't trust to it. And I don't even miss it.

I use containers explicitly for separation from main namespace for security; my nginx has no access to my network (except dns) for safety. Think twice.

I already influenced in containers/netavark ages ago and have given my effort for it, containers/netavark#597

In past, I had a fork of luci-app-dockerman for podman, it's still in one of my own package repositories https://github.com/oskarirauta/packages2/tree/main/luci/luci-app-upodman, but I haven't updated in years- and yet it wasn't complete, I lost my interest to it. Feel free to take advantage of it in your work; maybe it gives you a head start. If you update and complete it, consider making a PR to luci repo..

I do have a mini management interface, seen in that screenshot, upodman and a package for it.. And even a luci package here

It only has minimal management, like status details and possibilities to start/restart/stop containers.