Sysbox Runtime
The default docker container runtime is “runc”. Sysbox is an open-source and free container runtime, originally developed by Nestybox, to function as a drop-in replacement container runtime. Sysbox provides improved container isolation and support of VM workloads.
Kasm has the option of using Sysbox to allow Workspaces to run as root or with sudo privileges without comprimising the security of the Kasm Agent server. Using Sysbox with Kasm also provides the ability for Workspaces to run system-level software that is not normally possible in a containerized environment, such as systemd, buildx, and Docker.
If a Kasm user needs the ability to install software in a Workspace or use sudo (or root access), Sysbox is a recommended method to keep the Kasm Agent secure against container breakout. Please note that limitations apply.
The following Kasm Workspaces core images, and all variants built on top of those images, support a systemd init methodology:
Container |
Source Code |
|---|---|
This allows any container based off of these core images to be run with the Sysbox Runtime and init with systemd. When using Sysbox, containers behave much closer to Virtual Machines and allow users in a shared environment to run at root level in their Workspaces with a lower risk of escaping isolation. For additional security enhancements of using Sysbox with your Workspaces check out the Sysbox User Guide: Security.
Note
When using the Sysbox Runtime for a Kasm Workspaces session it may take longer for the Workspaces to be ready as compared to launching with native Docker init. This is due to the general overhead of using shiftfs along with a full systemd init chain needing to be run before the KasmVNC service is finally started.
Limitations of Kasm with Sysbox
These features are incompatible with Sysbox:
Persistent profiles
NVIDIA GPU support within Kasm Workspaces
Limited Storage mapping or Volume mapping support. No support between Workspaces using Sysbox and Workspaces not using Sysbox (see Using Storage Mapping with Sysbox for more information on this)
Installing Sysbox
Before getting started see the list of supported host operating systems here, and the official installation guide here. The purpose of this guide is not to replace the existing Sysbox methodology just review our recommended setup for using the sysbox-runc runtime.
Currently we recommend using an Ubuntu Jammy host running the 6.5 LTS kernel as this will have shiftfs integrated with the kernel. If you have a Linux kernel version between 5.15 and 6.3, refer to the Install shiftfs steps below. The minimum compatible Linux kernel for Sysbox is 5.15.
To ensure the necessary Kasm dependencies are installed correctly it is recommended to download and extract the Kasm installer. Once done, navigate to the kasm_release directory and run the install_dependencies.sh script. This will install all Kasm prerequisits after which you can Install Sysbox.
cd /tmp curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz tar -xf kasm_release_1.16.1.98d6fa.tar.gz sudo bash kasm_release/install_dependencies.sh
Install Sysbox Runtime
As mentioned previously this step should be performed after Docker is installed but before Kasm Workspaces is installed. Sysbox can be installed on top of a machine with an active Kasm Workspaces deployment, but we cannot guarantee containers will not be removed or deleted.
sudo apt-get install jq wget wget https://downloads.nestybox.com/sysbox/releases/v0.6.4/sysbox-ce_0.6.4-0.linux_amd64.deb sudo apt-get install ./sysbox-ce_0.6.4-0.linux_amd64.deb
Post installation confirm the Docker daemon was configured correctly:
cat /etc/docker/daemon.json { "runtimes": { "sysbox-runc": { "path": "/usr/bin/sysbox-runc" } }, "bip": "172.20.0.1/16", "default-address-pools": [ { "base": "172.25.0.0/16", "size": 24 } ] }
Configuring Workspaces to use Sysbox Runtime
Any Workspace that you would like to leverage the Sysbox Runtime on will need their runtime set to use systemd for init. This can be achieved with the following Docker Run Config Override (JSON):
{ "runtime": "sysbox-runc", "entrypoint": [ "/sbin/init" ], "user": 0 }
This setting can be Workspace specific under Admin > Workspaces > Edit > Docker Run Config Override (JSON) or set at a group level with Access Management > Groups > Edit > Settings tab > Add Setting > run_config.
Note
When using the Sysbox Runtime the Workspace will lose the ability to leverage an NVIDIA GPU using the NVIDIA Container Runtime. DRI3 using open source drivers will still function in a Sysbox Workspace, for more information on that please see the DRI3 documentation.
Enable sudo for users
Now that Sysbox is installed and the Workspace is using it as the designated runtime, it is time to allow sudo for the workspace. Within the Kasm Workspace configuration page, scroll down to the Docker Exec Config block and add the below JSON block. This code will run as root when the Workspace first launches, then it will install sudo, and add the kasm-user to the sudoers group. To read more about configuring a Kasm Workspace for sudo using Docker Exec, check the Installing and Configuring sudo via Docker Exec docs.
{ "first_launch":{ "user":"root", "cmd":"bash -c '/usr/bin/desktop_ready && apt-get update && apt-get install -y sudo && echo \"kasm-user ALL=(ALL) NOPASSWD: ALL\" >> /etc/sudoers'" } }
Using Storage Mapping with Sysbox
Important
When using a Storage Provider or Volume mapping configuration with Sysbox, attaching the same Storage Provider or Volume mapping to a non-Sysbox container will cause permission problems between Workspaces. The owner of the files and folders in a Sysbox container will be different than the owner of a non-Sysbox container. Read more below to understand why this happens and why it is important.
Using Kasm’s Storage Provider and/or Volume mapping with Sysbox introduces some differences in the way this feature normally works (for additional information on why, check out the Sysbox Documentation as well as their documentation on User Namespace ID Mapping).
When Kasm starts a Workspace container normally it runs the container as a user called kasm-user inside all of it’s Workspaces (refer to our Building Custom Images to see how we do this). This user has a UID (User ID) and GID (Group ID) of 1000. When Kasm maps Storage or Volume drives to a Workspace the UID/GID of 1000 is used to grant permission to the files and folders in the shared drive. Because of this fact, most of Kasm’s Storage Mapping documentation uses a config similar to the below JSON:
{ "driver" : "rclone", "driver_opts" : { "type" : "s3", "s3-provider" : "AWS", "s3-env-auth" : "false", "s3-region" : "us-east-1", "uid" : "1000", "gid" : "1000", "allow_other" : "true" } }
In the above JSON field, the "uid" : "1000" and "gid" : "1000" values are used to set the permissions on all of the files and folders in the share to correspond with the kasm-user UID and GID so the Workspace user has permission to view and modify files and folders in the mapped volume. To understand how Sysbox changes things it is important to get a good handle on how it is remapping UID and GID values and what Kasm does with those for file and folder permissions.
Sysbox changes permissions because it remaps the UID and GID values from the host to random, high values with no or very limited permissions on the host. Under the hood, the sysbox-runc runtime uses Docker’s User Namespaces to handle this remapping. This allows apparent root level access inside the container, while associating all container processes to a random high UID/GID value on the host. To see what this looks like on a host you can cat the /etc/passwd file to see what users exist and to see their associated UID and GID values.
$ cat /etc/passwd ... opc:x:1000:1000::/home/opc:/bin/sh ubuntu:x:1001:1001:Ubuntu:/home/ubuntu:/bin/bash kasm:x:1002:1002::/home/kasm:/bin/sh sysbox:x:1003:1003::/home/sysbox:/bin/false ...
On this host, there are 4 (four) users with login permssions and they have the UID values from 1000-1003 and GID values from 1000-1003. To see the Sysbox remapped UID/GID values you can cat the /etc/subuid and /etc/subgid files on your host after installing Sysbox.
$ cat /etc/subuid opc:100000:65536 ubuntu:165536:65536 kasm:231072:65536 sysbox:296608:65536
$ cat /etc/subgid opc:100000:65536 ubuntu:165536:65536 kasm:231072:65536 sysbox:296608:65536
Note
When using Sysbox CE, the only user responsible for running containers is the sysbox user. Thus, the only UID/GID values of importance in the /etc/subuid or /etc/subgid files are those associated with the sysbox user.
In these files it is possible to see that Sysbox has remapped the user sysbox with a host UID of 1003 and host GID of 1003 to the random high UID of 296608 and random high GID of 296608 and has given it a total of 65536 additional UID and GID values for any other required users inside its containers. Thus, from the host perspective, when a container runs using the sysbox-runc runtime, all processes and file or folder permissions will be associated with the UID and GID of 296608 on this host (this will likely be different on yours). From the container perspective, however, the user is root with a UID and GID of 0. Below we show running a container with the Sysbox runtime and get some information from it and from the host to demonstrate this process.
## Run a container using the `sysbox-runc` runtime ubuntu@kasm-agent:~$ docker run --runtime=sysbox-runc --name=sysbox-test --rm -d alpine tail -f /dev/null 2a70c22a9c3af66e3f4f76df3de9cf46dafed97ac7831953b0ebc3561eeeda91 ## Container view using uid_map ubuntu@kasm-agent:~$ docker exec sysbox-test cat /proc/self/uid_map 0 296608 65536 ## User info from within the container ubuntu@kasm-agent:~$ docker exec sysbox-test id uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) ## Host process info of `tail` command running inside the container ubuntu@kasm-agent:~$ ps -fC "tail -f /dev/null" UID PID PPID C STIME TTY TIME CMD 296608 171532 171503 0 21:35 ? 00:00:00 tail -f /dev/null
Now that we know the UID and GID of the container root user is 296608 we must figure out what the UID and GID is of the kasm-user used inside of every Kasm Workspace container. Previously we said that the UID and GID of the kasm-user inside a Kasm Workspace is 1000. So, to figure out how we need to modify the Storage or Volume mapping permissions values, we must take the Sysbox root UID of 296608 and add 1000 to it to get the host UID of the kasm-user.
Get kasm-user UID for Storage/Volume Mapping:
Attention
Aside from the kasm-user UID (1000), the values above are the ones the Sysbox service used for my deployment. Yours will likely be different!
Now that we have the kasm-user UID and GID values we can update our Storage or Volume mapping values so that our file shares work appropriately with these users.
{ "driver" : "rclone", "driver_opts" : { "type" : "s3", "s3-provider" : "AWS", "s3-env-auth" : "false", "s3-region" : "us-east-1", "uid" : "297608", "gid" : "297608", "allow_other" : "true" } }
Install shiftfs
Application “shiftfs” is a kernel filesystem for the Linux kernel. It provides easier uid/gid-shifting for containers. Shiftfs is a dependency of Sysbox. Linux kernels 5.15.x and later have shiftfs preinstalled.
Installing shiftfs
While Kernels 5.15-6.3 in Ubuntu Jammy do support shiftfs it is best to install an up to date version for performance reasons.
sudo apt-get update
sudo apt-get install -y make dkms git wget
git clone -b k5.16 https://github.com/toby63/shiftfs-dkms.git shiftfs-k516
cd shiftfs-k516/
./update1
sudo make -f Makefile.dkms
modinfo shiftfs
You should see output similar to:
filename: /lib/modules/5.15.0-1031-aws/updates/dkms/shiftfs.ko
license: GPL v2
description: id shifting filesystem
author: Christian Brauner <christian.brauner@ubuntu.com>
author: Seth Forshee <seth.forshee@canonical.com>
author: James Bottomley
alias: fs-shiftfs
At this point shiftfs is ready, but best to reboot the machine before continuing sudo reboot.