Podman
Introduction
Podman is an OCI compliant container management tool that offers similar features like Docker for managing containers.
One of the best features of podman is its ability to run rootless containers. A rootless container is a concept of running and managing containers without root privileges (Normal user). From a security standpoint, rootless containers add an additional layer of security by not allowing root access even if the container gets compromised by an attacker. You can read about the benefits of rootless containers here
Note
Docker also supports rootless mode with some limitations. You can read about it in this documentation
Podman is also daemonless (unlike docker), meaning it doesn’t have a daemon and interacts directly with the runc (runs containers based on OCI specification). Toward the end of the article, I added the difference between Docker and Podman.
Also, let’s say there are two users in Linux. user-a and user-b. The containers created by user-a using podman cannot be modified by user-b and vice versa.
Another interesting and advanced feature of podman is running containers in Pods. Similar to Kubernetes pods, you can create multi-container pods locally using Podman. You can export the podman pod as a Kubernetes manifest and use a Kubernetes pod manifest to deploy and podman pod.
Podman |
Docker |
|---|---|
Podman is Daemonless |
Docker has a daemon (containerd ). The docker CLI interacts with the daemon to manage containers. |
Podman interacts with the Linux kernel directly through runc |
Docker daemon owns all the child processes of running containers |
Podman can deploy pods with multiple containers. The same pod manifest can be used in Kubernetes. Also, you can deploy a Kubernetes pod manifest as a Podman pod. |
There is no concept of a pod in Docker |
Can run rootless containers without any additional configurations. You can run containers with root or non-privileged users. |
Docker rootless mode requires additional configurations. |
Installation
# Update your packages
sudo apt-get update && sudo apt-get upgrade
# Install Podman
sudo apt-get install podman
From there usage is as normal
Usage
# Create Podman Machine
podman machine init
# Start Podman
podman machine start
# Verify Info
podman info
Similar to docker, most commands are the same
# Start Container
podman start
# Stop Container
podman stop
# View Processes
podman ps
# Remove Container
podman rm
Registries
We can also grab images from any repository; Github, Docker, Google, Redhat, Kubernates
From Podman Add Registry
# System-Wide configuration
nano /etc/containers/registries.conf
# User-Specific configuration
nano $HOME/.config/containers/registries.conf
# An example would be
[[registry]]
location=quay.io
# Or
[[registry]]
location="my-registry.tld"
username="username"
password="password"
This allows us to have anything accessible for us to pull..
Permissions
Podman permissions seem tricky at first, but let’s try to understand them together..
Podman utilises something called SELinux which is a kernel addon created by the NSA for high security settings, although we don’t need this additional security, it is very good to have considering containers are supposed to be secure by design.
If you read into how it works, you will slowly start to understand that podman mimics the systems permissions in a strange way, as everything inside the container needs to be run as root under podmans control, the outside permissions need to allow this
Below is a table that somewhat explains this.
You run podman as.. |
With containerised process running as.. |
The actual UID visible on the host is… |
|---|---|---|
root |
root |
0 |
root |
non-root |
The UID of the user running the process inside the container |
non-root |
root |
Your UID |
non-root |
non-root |
A non-root UID |
All rootless containers must be run in the same user namespace. If they are not, some things (like sharing the network namespace from another container) would be impossible.
By using the same user namespace, your containers can share resources with each other, without needing to ask for root privileges.
It uses this user namespace to mount filesystems, or run a container which accesses more than one user ID (UID) or group ID (GID).
This mapping is fine for most situations, except when the container needs to be able to share something with the host, like a volume.
But - here’s the important thing:
When the container runs, any volumes which are shared with it, will appear inside the user namespace as owned by root/root. Because the mapping will map your UID on the host (e.g. 1000) as root (0) in the container.
Examples
We have gone through and figured out roughly how to get these permissions working nicely in the container.
The examples provided are for single run commands, but include the PUID and PGID
podman run -d \
--name=nextcloud \
-e PUID=7000 \
-e PGID=7000 \
-e TZ=Europe/London \
-p 443:443 \
-v /home/nextcloud/config:/config \
-v /home/nextcloud/data:/data \
--restart unless-stopped \
lscr.io/linuxserver/nextcloud:latest
Here the example is for nextcloud, we’ve created a user nextcloud on the server, the UID are set on the user when created.
su nextcloud
mkdir config
mkdir data
#Run the podman command here
Since the directories are created under the nextcloud user, they have control and access..
Podman then hooks onto this in a sense, leverages the users permissions then elevates to root within the container.
Referring to the table from before, this user is non-root, so podman will be root.
Command Structure
From what I have seen there are different ways to structure a run command, however they are all Similar
podman run -d \
--name=<name> \
-e PUID=<uid> \
-e PGID=<gid> \
-e <EXTRA> \
-p <port number> \
-v <volume/directory> \
--restart <option> \
<repository link>
Going through this line by line, we have the container name which should be obvious
Next we have PUID and PGID, these are what allow the container to hook onto the user and elevate root inside the container
Next we have an optional extra, this can be another environment variable like timezone or something else like gpu support
Now onto the port number, the way this works is simple. There’s 000:000. The number before the brackets is the port on our local machine the port after the brackets is what the container will use. So for web servers that’ll be :80, :8080, :443, etc.. The idea being you can run multiple services on one or several ports with simple port forwarding.
After that is the volume, we set these up so we can tell the container where to store data or configuration files, similar to how the port forwarding works We have /home/$USER/dev:/dev. So within the container it requires a dev folder, we create a directory on our host machine and link the two together. This is why setting UID permissions is important since without them, we would not be able to communicate or do anything within this volume.
Next is just the restart option, if you want to use always, unless-stopped, never As you can guess, always is just always looping, unless-stopped is to restart it if for some reason it crashes and stops, never just means if it ever stops, don’t restart it.
Podman pods
What are pods?
The pod concept was introduced by the Kubernetes project. First, a pod itself has no application in it, but it can manage all the resources like a container does. A pod basically wraps containers, storage resources, and an ephemeral network identity together as a single unit.
We’ve gone over how normal containers are created, now to put it into action with pods, we can do this
Pods in containers
# Create a pod
$ podman pod create -p 8080:80 --name pod01
# List pods
$ podman pod ls
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
c5d90af17283 pod Created 8 seconds ago 9b720f35d62d 1
# Start a container in the pod
$ podman container run -d --name container01 \
--pod pod01 docker.io/library/httpd
# List containers
$ podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b720f35d62d k8s.gcr.io/pause:3.2 27 seconds ago Up 6 seconds ago 0.0.0.0:8080->80/tcp c5d90af17283-infra
111e6c72b994 docker.io/library/httpd httpd-foreground 5 seconds ago Up 6 seconds ago 0.0.0.0:8080->80/tcp container01
As you can see, there is another container running, based on the k8s.gcr.io/pause image. Port bindings, cgroup values, and kernel namespaces are assigned to this infra container. Once the pod is created these attributes are assigned to the infra container and cannot be changed.
Multiple container in a pod
You can also have multiple containers, in the same pod. The containers will share some resources and therefore are behaving as “one service”.
Let’s add two more containers to the pod.
# Start a second container in the pod
$ podman container run -d --name container02 --pod pod01 docker.io/library/mariadb
# Start a third container in the pod
$ podman container run -d --name container03 --pod pod01 docker.io/library/redis
# List pods
$ podman pod ls
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
c5d90af17283 pod01 Degraded About a minute ago 9b720f35d62d 4
# List containers
$ podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9b720f35d62d k8s.gcr.io/pause:3.2 2 minutes ago Up About a minute ago 0.0.0.0:8080->80/tcp c5d90af17283-infra
111e6c72b994 docker.io/library/httpd httpd-foreground About a minute ago Up About a minute ago 0.0.0.0:8080->80/tcp container01
01713e4871f3 docker.io/library/redis:latest redis-server 22 seconds ago Up 22 seconds ago 0.0.0.0:8080->80/tcp container03
Benefits
Using pods can be very useful when sharing resources across containers.
Often one needs containers to communicate with each other. A web server should be able to communicate with a Redis server or a database for example (fortunately our example above is exactly about this). With 3 containers in the same pod, all the services are working, as if they are the same machine, and you can use “localhost:PORT” to communicate.
The web server is published and reachable via 8080/TCP from outside the host, but the database and Redis server are not. But, the web server can use them via “localhost:3306” and “localhost:6379” without publishing them explicitly or fiddling with IP addresses.
This is also very useful for sidecar containers, that collect metrics, logs, etc. and push them to a central instance.
Limitations
Well, there are some limitations, that you need to be aware of.
First, you cannot add additional published ports to the already created pods. In case you need to add more ports, you need to re-create the pod, and its containers.
Furthermore, you need to be aware, that this behavior may lead to security issues. The PostgreSQL container image for example sees all traffic from 127.0.0.1 as “secure” traffic. Since all containers in the same pod are “the same machine”, this may have impacts to your security concepts.
Lastly, you need to be aware, that you cannot start multiple containers with the same exposed port in a Pod. For example starting two containers based on docker.io/library/httpd is not possible.
Podman Kube Files
What are kubes?
Whenever Podman developers talk about its future, they always mention one topic—making it easier to test workloads with Podman and deploy them into Kubernetes. The primary way users jump between Podman and Kubernetes is by using Podman’s generate kube and play kube subcommands. As the names imply, generate kube creates a YAML description of a Podman pod or container to run in Kubernetes. The play kube subcommand allows you to run Podman pods based on a Kubernetes YAML file.
Build images with play kube
The new podman play kube feature looks for a directory with the same name as the image used in the YAML file. If that directory exists and there is a Containerfile or Dockerfile present in that directory, Podman builds the container image.
For example, a sample YAML file that has Apache and PHP could look like
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.0.0-dev
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-09-20T17:40:19Z"
labels:
app: php
name: php
spec:
containers:
- args:
- apache2-foreground
command:
- docker-php-entrypoint
env:
- name: PATH
value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
- name: TERM
value: xterm
- name: container
...
- name: PHP_EXTRA_BUILD_DEPS
value: apache2-dev
- name: APACHE_ENVVARS
value: /etc/apache2/envvars
image: php-7.2-apache-mysqli:latest
name: apache
ports:
- containerPort: 80
hostPort: 8080
protocol: TCP
resources: {}
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- CAP_MKNOD
- CAP_NET_RAW
- CAP_AUDIT_WRITE
privileged: false
readOnlyRootFilesystem: false
seLinuxOptions: {}
tty: true
workingDir: /var/www/html
dnsConfig: {}
restartPolicy: Never
status: {}
Notice in the YAML file that the container image is referred to as php-7.2-apache-mysqli:latest. If there is a Containerfile in that directory and the image is not in the image store, Podman builds the image.
Now look at an example where you want to use docker.io/library/php:7.2-apache but the PHP extension for mysqli is not installed or enabled. Create a Containerfile inside the php-7.2-apache-mysqli directory. To give some perspective, here’s the directory layout
├── mariadb-conf
│ ├── Containerfile
│ └── my.cnf
├── php-7.2-apache-mysqli
│ ├── Containerfile
│ └── index.php
└── php.yaml
And the contents of the Containerfile are
$ cat php-7.2-apache-mysqli/Containerfile
FROM docker.io/library/php:7.2-apache
RUN docker-php-ext-install mysqli
COPY index.php /var/www/html/index.php
And now, execute the play kube command, citing the YAML file
$ podman play kube php.yaml
--> /home/baude/myproject/php-7.2-apache-mysqli
STEP 1/3: FROM docker.io/library/php:7.2-apache
STEP 2/3: RUN docker-php-ext-install mysqli
Configuring for:
PHP Api Version: 20170718
Zend Module Api No: 20170718
Zend Extension Api No: 320170718
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking for cc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
…
Build complete.
Don't forget to run 'make test'.
Installing shared extensions: /usr/local/lib/php/extensions/no-debug-non-zts-20170718/
Installing header files: /usr/local/include/php/
find . -name \*.gcno -o -name \*.gcda | xargs rm -f
find . -name \*.lo -o -name \*.o | xargs rm -f
find . -name \*.la -o -name \*.a | xargs rm -f
find . -name \*.so | xargs rm -f
find . -name .libs -a -type d|xargs rm -rf
rm -f libphp.la modules/* libs/*
STEP 3/3: COPY index.php /var/www/html/index.php
COMMIT php-7.2-apache-mysqli:latest
--> 096882adf84
Successfully tagged localhost/php-7.2-apache-mysqli:latest
096882adf845274f8c6546cf52a77c7fc78b3fa20c659cfc2b73753972bd5f90
Pod:
d8774760bc3a0bdb1b405c57880f3872b0a71f3434b8d4ab6d8d18c8e6e44ffa
Container:
25162b2da0e2b9078c69404f6d6e1250f6ee4fc032b47564f9bbb971d838cf29
Tear down pods with play kube
The podman play kube command can create and run multiple pods with multiple containers in the pods. Managing the pods that play kube creates and runs has always been a manual process using Podman’s pod commands.
Revisiting the PHP container example above, once the play kube command executes, you can observe a running pod with two containers, including the pod’s infra container
$ podman pod ps
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
cc97c8c8a07d php Running 20 hours ago bd22e1434d3a 2
$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bd22e1434d3a k8s.gcr.io/pause:3.5 22 hours ago Up 22 hours ago 0.0.0.0:8080->80/tcp cc97c8c8a07d-infra
b97893b79bb9 localhost/php-7.2-apache-mysqli:latest apache2-foregroun... 22 hours ago Up 22 hours ago 0.0.0.0:8080->80/tcp php-greatnapier
Suppose you want to stop this pod. You can “replay” the YAML file by adding the –down flag
$ podman play kube --down php.yaml
Pods stopped:
cc97c8c8a07db0f26114022a71ee59771134244a8b465147a513b28f9b7d171b
Pods removed:
cc97c8c8a07db0f26114022a71ee59771134244a8b465147a513b28f9b7d171b
$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ podman pod ps
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
The pod is stopped and then removed. Confirm the pod’s containers are also removed with podman ps -a or podman pod ps.
Common Issues
Following are some of the errors and issues I faced while doing hands-on with Podman.
Port Mapping Error
If you try to map privileged ports you might get the following error. Always use a nonprivileged port if you want to run Podman as a non-root user.
Error: error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]
If you don’t have runc installed, you might get the following error.
Error: default OCI runtime "runc" not found: invalid argument
Image Errors
If you don’t specify the correct image name, Podman will throw the following error.
Trying to pull quay.io/busybox...
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
It normally happens with quay.io because, unlike docker, for quay.io , you should specify the image name correctly.
For example, podman pull busybox, will throw the error. But, podman pull quay/busybox works without any error.
Pod Errors
If you try to add a container with ports that were not added during the pod creation, you will get the following error.
Error: invalid config provided: published or exposed ports must be defined when the pod is created: network cannot be configured when it is shared with a pod
If you try to delete a pod with running containers, you will get the following error. First, you need to stop all the containers and then delete the pod.
Error: pod 9e31de31950664702f21 has containers that are not ready to be removed: cannot remove container 3d10007e844a5aea3c7805fb0ee986b0b4c2cedd66c0996d0bff9f53ba1c0b57 as it is running - running or paused containers cannot be removed without force: container state improper
Sometimes you might deploy containers using a specific user and if you try to list the containers with a different user or with sudo, you will get the following error.
Error: no pod with name or ID webserver found: no such pod
Sources of Information
Here are my Sources