Podman

Introduction

Podman is an OCI compliant container management tool that offers similar features like Docker for managing containers.

One of the best features of podman is its ability to run rootless containers. A rootless container is a concept of running and managing containers without root privileges (Normal user). From a security standpoint, rootless containers add an additional layer of security by not allowing root access even if the container gets compromised by an attacker. You can read about the benefits of rootless containers here

Note

Docker also supports rootless mode with some limitations. You can read about it in this documentation

Podman is also daemonless (unlike docker), meaning it doesn’t have a daemon and interacts directly with the runc (runs containers based on OCI specification). Toward the end of the article, I added the difference between Docker and Podman.

Also, let’s say there are two users in Linux. user-a and user-b. The containers created by user-a using podman cannot be modified by user-b and vice versa.

Another interesting and advanced feature of podman is running containers in Pods. Similar to Kubernetes pods, you can create multi-container pods locally using Podman. You can export the podman pod as a Kubernetes manifest and use a Kubernetes pod manifest to deploy and podman pod.

Docker Vs Podman
Podman	Docker
Podman is Daemonless	Docker has a daemon (containerd ). The docker CLI interacts with the daemon to manage containers.
Podman interacts with the Linux kernel directly through runc	Docker daemon owns all the child processes of running containers
Podman can deploy pods with multiple containers. The same pod manifest can be used in Kubernetes. Also, you can deploy a Kubernetes pod manifest as a Podman pod.	There is no concept of a pod in Docker
Can run rootless containers without any additional configurations. You can run containers with root or non-privileged users.	Docker rootless mode requires additional configurations.

Installation

# Update your packages
sudo apt-get update && sudo apt-get upgrade
# Install Podman
sudo apt-get install podman

From there usage is as normal

Usage

# Create Podman Machine
podman machine init
# Start Podman
podman machine start
# Verify Info
podman info

Similar to docker, most commands are the same

# Start Container
podman start
# Stop Container
podman stop
# View Processes
podman ps
# Remove Container
podman rm

Registries

We can also grab images from any repository; Github, Docker, Google, Redhat, Kubernates

From Podman Add Registry

# System-Wide configuration
nano /etc/containers/registries.conf

# User-Specific configuration
nano $HOME/.config/containers/registries.conf

# An example would be
[[registry]]
location=quay.io

# Or

[[registry]]
location="my-registry.tld"
username="username"
password="password"

This allows us to have anything accessible for us to pull..

Permissions

Podman permissions seem tricky at first, but let’s try to understand them together..

Podman utilises something called SELinux which is a kernel addon created by the NSA for high security settings, although we don’t need this additional security, it is very good to have considering containers are supposed to be secure by design.

If you read into how it works, you will slowly start to understand that podman mimics the systems permissions in a strange way, as everything inside the container needs to be run as root under podmans control, the outside permissions need to allow this

Below is a table that somewhat explains this.

From Podman Rootless Volumes

The table below shows four possible rootless/rootful operating modes of Podman
You run podman as..	With containerised process running as..	The actual UID visible on the host is…
root	root	0
root	non-root	The UID of the user running the process inside the container
non-root	root	Your UID
non-root	non-root	A non-root UID

All rootless containers must be run in the same user namespace. If they are not, some things (like sharing the network namespace from another container) would be impossible.

By using the same user namespace, your containers can share resources with each other, without needing to ask for root privileges.

It uses this user namespace to mount filesystems, or run a container which accesses more than one user ID (UID) or group ID (GID).

This mapping is fine for most situations, except when the container needs to be able to share something with the host, like a volume.

But - here’s the important thing:

When the container runs, any volumes which are shared with it, will appear inside the user namespace as owned by root/root. Because the mapping will map your UID on the host (e.g. 1000) as root (0) in the container.

Examples

We have gone through and figured out roughly how to get these permissions working nicely in the container.

The examples provided are for single run commands, but include the PUID and PGID

podman run -d \
--name=nextcloud \
-e PUID=7000 \
-e PGID=7000 \
-e TZ=Europe/London \
-p 443:443 \
-v /home/nextcloud/config:/config \
-v /home/nextcloud/data:/data \
--restart unless-stopped \
lscr.io/linuxserver/nextcloud:latest

Here the example is for nextcloud, we’ve created a user nextcloud on the server, the UID are set on the user when created.

su nextcloud
mkdir config
mkdir data

#Run the podman command here

Since the directories are created under the nextcloud user, they have control and access..

Podman then hooks onto this in a sense, leverages the users permissions then elevates to root within the container.

Referring to the table from before, this user is non-root, so podman will be root.

Command Structure

From what I have seen there are different ways to structure a run command, however they are all Similar

podman run -d \
--name=<name> \
-e PUID=<uid> \
-e PGID=<gid> \
-e <EXTRA> \
-p <port number> \
-v <volume/directory> \
--restart <option> \
<repository link>

Going through this line by line, we have the container name which should be obvious

Next we have PUID and PGID, these are what allow the container to hook onto the user and elevate root inside the container

Next we have an optional extra, this can be another environment variable like timezone or something else like gpu support

Now onto the port number, the way this works is simple. There’s 000:000. The number before the brackets is the port on our local machine the port after the brackets is what the container will use. So for web servers that’ll be :80, :8080, :443, etc.. The idea being you can run multiple services on one or several ports with simple port forwarding.

After that is the volume, we set these up so we can tell the container where to store data or configuration files, similar to how the port forwarding works We have /home/$USER/dev:/dev. So within the container it requires a dev folder, we create a directory on our host machine and link the two together. This is why setting UID permissions is important since without them, we would not be able to communicate or do anything within this volume.

Next is just the restart option, if you want to use always, unless-stopped, never As you can guess, always is just always looping, unless-stopped is to restart it if for some reason it crashes and stops, never just means if it ever stops, don’t restart it.

Podman pods

What are pods?

The pod concept was introduced by the Kubernetes project. First, a pod itself has no application in it, but it can manage all the resources like a container does. A pod basically wraps containers, storage resources, and an ephemeral network identity together as a single unit.

We’ve gone over how normal containers are created, now to put it into action with pods, we can do this

Pods in containers

# Create a pod
$ podman pod create -p 8080:80 --name pod01

# List pods
$ podman pod ls
POD ID        NAME    STATUS   CREATED        INFRA ID      # OF CONTAINERS
c5d90af17283  pod   Created  8 seconds ago  9b720f35d62d  1

# Start a container in the pod
$ podman container run -d --name container01 \
  --pod pod01 docker.io/library/httpd

# List containers
$ podman container ls
CONTAINER ID  IMAGE                                             COMMAND               CREATED         STATUS            PORTS                 NAMES
9b720f35d62d  k8s.gcr.io/pause:3.2                                                    27 seconds ago  Up 6 seconds ago  0.0.0.0:8080->80/tcp  c5d90af17283-infra
111e6c72b994  docker.io/library/httpd                           httpd-foreground      5 seconds ago   Up 6 seconds ago  0.0.0.0:8080->80/tcp  container01

As you can see, there is another container running, based on the k8s.gcr.io/pause image. Port bindings, cgroup values, and kernel namespaces are assigned to this infra container. Once the pod is created these attributes are assigned to the infra container and cannot be changed.

Multiple container in a pod

You can also have multiple containers, in the same pod. The containers will share some resources and therefore are behaving as “one service”.

Let’s add two more containers to the pod.

# Start a second container in the pod
$ podman container run -d --name container02 --pod pod01 docker.io/library/mariadb

# Start a third container in the pod
$ podman container run -d --name container03 --pod pod01 docker.io/library/redis

# List pods
$ podman pod ls
POD ID        NAME    STATUS    CREATED             INFRA ID      # OF CONTAINERS
c5d90af17283  pod01   Degraded  About a minute ago  9b720f35d62d  4

# List containers
$ podman container ls
CONTAINER ID  IMAGE                                             COMMAND               CREATED             STATUS                 PORTS                 NAMES
9b720f35d62d  k8s.gcr.io/pause:3.2                                                    2 minutes ago       Up About a minute ago  0.0.0.0:8080->80/tcp  c5d90af17283-infra
111e6c72b994  docker.io/library/httpd                           httpd-foreground      About a minute ago  Up About a minute ago  0.0.0.0:8080->80/tcp  container01
01713e4871f3  docker.io/library/redis:latest                    redis-server          22 seconds ago      Up 22 seconds ago      0.0.0.0:8080->80/tcp  container03

Benefits

Using pods can be very useful when sharing resources across containers.

Often one needs containers to communicate with each other. A web server should be able to communicate with a Redis server or a database for example (fortunately our example above is exactly about this). With 3 containers in the same pod, all the services are working, as if they are the same machine, and you can use “localhost:PORT” to communicate.

The web server is published and reachable via 8080/TCP from outside the host, but the database and Redis server are not. But, the web server can use them via “localhost:3306” and “localhost:6379” without publishing them explicitly or fiddling with IP addresses.

This is also very useful for sidecar containers, that collect metrics, logs, etc. and push them to a central instance.

Limitations

Well, there are some limitations, that you need to be aware of.

First, you cannot add additional published ports to the already created pods. In case you need to add more ports, you need to re-create the pod, and its containers.

Furthermore, you need to be aware, that this behavior may lead to security issues. The PostgreSQL container image for example sees all traffic from 127.0.0.1 as “secure” traffic. Since all containers in the same pod are “the same machine”, this may have impacts to your security concepts.

Lastly, you need to be aware, that you cannot start multiple containers with the same exposed port in a Pod. For example starting two containers based on docker.io/library/httpd is not possible.

Podman Kube Files

What are kubes?

Whenever Podman developers talk about its future, they always mention one topic—making it easier to test workloads with Podman and deploy them into Kubernetes. The primary way users jump between Podman and Kubernetes is by using Podman’s generate kube and play kube subcommands. As the names imply, generate kube creates a YAML description of a Podman pod or container to run in Kubernetes. The play kube subcommand allows you to run Podman pods based on a Kubernetes YAML file.

Build images with play kube

The new podman play kube feature looks for a directory with the same name as the image used in the YAML file. If that directory exists and there is a Containerfile or Dockerfile present in that directory, Podman builds the container image.

For example, a sample YAML file that has Apache and PHP could look like

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.0.0-dev
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2021-09-20T17:40:19Z"
  labels:
      app: php
  name: php
spec:
  containers:
  - args:
      - apache2-foreground
      command:
      - docker-php-entrypoint
      env:
      - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      - name: TERM
      value: xterm
      - name: container
      ...
      - name: PHP_EXTRA_BUILD_DEPS
      value: apache2-dev
      - name: APACHE_ENVVARS
      value: /etc/apache2/envvars
      image: php-7.2-apache-mysqli:latest
      name: apache
      ports:
      - containerPort: 80
      hostPort: 8080
      protocol: TCP
      resources: {}
      securityContext:
      allowPrivilegeEscalation: true
      capabilities:
      drop:
      - CAP_MKNOD
      - CAP_NET_RAW
      - CAP_AUDIT_WRITE
      privileged: false
      readOnlyRootFilesystem: false
      seLinuxOptions: {}
      tty: true
      workingDir: /var/www/html
  dnsConfig: {}
  restartPolicy: Never
status: {}

Notice in the YAML file that the container image is referred to as php-7.2-apache-mysqli:latest. If there is a Containerfile in that directory and the image is not in the image store, Podman builds the image.

Now look at an example where you want to use docker.io/library/php:7.2-apache but the PHP extension for mysqli is not installed or enabled. Create a Containerfile inside the php-7.2-apache-mysqli directory. To give some perspective, here’s the directory layout

├── mariadb-conf
│   ├── Containerfile
│   └── my.cnf
├── php-7.2-apache-mysqli
│   ├── Containerfile
│   └── index.php
└── php.yaml

And the contents of the Containerfile are

$ cat php-7.2-apache-mysqli/Containerfile
FROM docker.io/library/php:7.2-apache
RUN docker-php-ext-install mysqli
COPY index.php /var/www/html/index.php

And now, execute the play kube command, citing the YAML file

$ podman play kube php.yaml
-->  /home/baude/myproject/php-7.2-apache-mysqli
STEP 1/3: FROM docker.io/library/php:7.2-apache
STEP 2/3: RUN docker-php-ext-install mysqli
Configuring for:
PHP Api Version:      20170718
Zend Module Api No:   20170718
Zend Extension Api No:   320170718
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking for cc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
…
Build complete.
Don't forget to run 'make test'.

Installing shared extensions:         /usr/local/lib/php/extensions/no-debug-non-zts-20170718/
Installing header files:              /usr/local/include/php/
find . -name \*.gcno -o -name \*.gcda | xargs rm -f
find . -name \*.lo -o -name \*.o | xargs rm -f
find . -name \*.la -o -name \*.a | xargs rm -f
find . -name \*.so | xargs rm -f
find . -name .libs -a -type d|xargs rm -rf
rm -f libphp.la       modules/* libs/*
STEP 3/3: COPY index.php /var/www/html/index.php
COMMIT php-7.2-apache-mysqli:latest
--> 096882adf84
Successfully tagged localhost/php-7.2-apache-mysqli:latest
096882adf845274f8c6546cf52a77c7fc78b3fa20c659cfc2b73753972bd5f90
Pod:
d8774760bc3a0bdb1b405c57880f3872b0a71f3434b8d4ab6d8d18c8e6e44ffa
Container:
25162b2da0e2b9078c69404f6d6e1250f6ee4fc032b47564f9bbb971d838cf29

Tear down pods with play kube

The podman play kube command can create and run multiple pods with multiple containers in the pods. Managing the pods that play kube creates and runs has always been a manual process using Podman’s pod commands.

Revisiting the PHP container example above, once the play kube command executes, you can observe a running pod with two containers, including the pod’s infra container

$ podman pod ps
POD ID        NAME            STATUS          CREATED         INFRA ID        # OF CONTAINERS
cc97c8c8a07d  php             Running         20 hours ago  bd22e1434d3a  2
$ podman ps -a
CONTAINER ID  IMAGE                                   COMMAND                 CREATED         STATUS          PORTS                   NAMES
bd22e1434d3a  k8s.gcr.io/pause:3.5                                            22 hours ago  Up 22 hours ago  0.0.0.0:8080->80/tcp  cc97c8c8a07d-infra
b97893b79bb9  localhost/php-7.2-apache-mysqli:latest  apache2-foregroun...  22 hours ago  Up 22 hours ago  0.0.0.0:8080->80/tcp  php-greatnapier

Suppose you want to stop this pod. You can “replay” the YAML file by adding the –down flag

$ podman play kube --down php.yaml
Pods stopped:
cc97c8c8a07db0f26114022a71ee59771134244a8b465147a513b28f9b7d171b
Pods removed:
cc97c8c8a07db0f26114022a71ee59771134244a8b465147a513b28f9b7d171b
$ podman ps -a
CONTAINER ID  IMAGE           COMMAND         CREATED         STATUS          PORTS           NAMES
$ podman pod ps
POD ID      NAME        STATUS      CREATED     INFRA ID    # OF CONTAINERS

The pod is stopped and then removed. Confirm the pod’s containers are also removed with podman ps -a or podman pod ps.

Common Issues

Following are some of the errors and issues I faced while doing hands-on with Podman.

Port Mapping Error

If you try to map privileged ports you might get the following error. Always use a nonprivileged port if you want to run Podman as a non-root user.

Error: error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]

If you don’t have runc installed, you might get the following error.

Error: default OCI runtime "runc" not found: invalid argument

Image Errors

If you don’t specify the correct image name, Podman will throw the following error.

Trying to pull quay.io/busybox...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"

It normally happens with quay.io because, unlike docker, for quay.io , you should specify the image name correctly.

For example, podman pull busybox, will throw the error. But, podman pull quay/busybox works without any error.

Pod Errors

If you try to add a container with ports that were not added during the pod creation, you will get the following error.

Error: invalid config provided: published or exposed ports must be defined when the pod is created: network cannot be configured when it is shared with a pod

If you try to delete a pod with running containers, you will get the following error. First, you need to stop all the containers and then delete the pod.

Error: pod 9e31de31950664702f21 has containers that are not ready to be removed: cannot remove container 3d10007e844a5aea3c7805fb0ee986b0b4c2cedd66c0996d0bff9f53ba1c0b57 as it is running - running or paused containers cannot be removed without force: container state improper

Sometimes you might deploy containers using a specific user and if you try to list the containers with a different user or with sudo, you will get the following error.

Error: no pod with name or ID webserver found: no such pod

Sources of Information

Here are my Sources

Source 1

Source 2

Source 3