---
myst:
  html_meta:
    "description lang=en": "GPU accelerated workloads and rendering with Kasm Workspaces configuration guide."
    "keywords": "Kasm, How to, How-to, NVIDIA, GPU, Transcoding, 3D, Graphics, Rendering, Blender, Gaming"
    "property=og:locale": "en_US"
---
```{title} GPU Acceleration
```

# GPU Acceleration

```{note}
This document outlines our officially supported GPU integration using the Nvidia container runtime. With a single server deployment it is also possible to use AMD/Intel GPUs on the host by manually configuring Workspace settings. See [Manual Intel or AMD GPU configuration](manual_intel_amd.md) for more information.
```

Kasm Workspaces supports passing GPUs through to container sessions. GPUs can be used for development of machine learning, data science, video encoding, graphics acceleration, and a number of other reasons. All uses of GPUs will require special host configurations. This guide will walk through preparing a bare VM with the default installation of Ubuntu 20.04LTS for GPU acceleration within Kasm Workspaces.

If Kasm is a multi-server deployment, these steps need to be performed on each Agent component only. If you are installing Kasm Workspaces on a single server, these steps would only need to be performed on that single instance.

```{caution}
The security of multi-tenant container GPU accelation is not well established. This feature should be used with caution and with full understanding of the potential security implications. These features are being provided for by the GPU manufacturer and are merely being orchestrated by Kasm Workspaces. Kasm Technologies Inc provides no warranty of the security of such features provided by GPU vendors. We recommend that GPUs not be shared between different security contexts and/or boundaries.
```

## Installing GPU Driver

Follow the guide bellow for the the appropriate GPU vendor.

```{eval-rst}
.. tabs::

    .. tab:: NVIDIA

        In this guide we use an AWS EC2 instance type of g4dn.xlarge with 4 CPUs, 16GB of RAM, and 1 NVIDIA Tesla T4 GPU. Any GPU supported by the `NVIDIA Container Toolkit <https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html>`_ should work. We used a vanilla Ubuntu 20.04LTS AMI, so these steps should not assume anything unique to AWS.

        .. code-block:: bash

            # update OS
            sudo apt-get update
            sudo apt-get upgrade -y

            # install dependencies for nvidia driver
            sudo apt-get upgrade -y linux-aws
            sudo apt-get install -y gcc make linux-headers-$(uname -r)

            # blacklist kernel modules that might interfere
            cat << EOF | sudo tee --append /etc/modprobe.d/blacklist.conf
            blacklist vga16fb
            blacklist nouveau
            blacklist rivafb
            blacklist nvidiafb
            blacklist rivatv
            EOF
            echo 'GRUB_CMDLINE_LINUX="rdblacklist=nouveau"' | sudo tee --append /etc/default/grub
            sudo update-grub

            # download latest driver, you may use a different method of obtaining the driver, this method uses the AWS CLI
            sudo apt install -y awscli
            aws s3 cp --no-sign-request  --recursive s3://ec2-linux-nvidia-drivers/latest/ .
            chmod +x NVIDIA-Linux-x86_64*.run

            # install
            sudo /bin/sh ./NVIDIA-Linux-x86_64*.run -s

            # check that card0 and renderD128 exist
            ls /dev/dri

            # check that nvidia-smi is working, it should report the driver and cuda version and card type
            nvidia-smi
```

## Install Kasm Workspaces

Installing Kasm Workspaces automatically installs Docker, if it is not already installed. So this step must be completed prior to installing the GPU container toolkit described in the next step. If you are installing Kasm Workspaces on a single server, follow the [standard installation guide](../install/single_server_install.md). If you are using a multi-server deployment for scalability and redundance, follow the Agent Role installation steps of the [Multi Server Guide](../install/multi_server_install.md).

## Installing Container Toolkit

To use the GPU(s) inside containers, you need to install the GPU vendor's container toolkit.

```{eval-rst}
.. tabs::

    .. tab:: NVIDIA

        .. code-block:: bash

            # Install the NVIDIA Container Toolkit
            distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
            sudo apt-get update
            sudo apt-get install -y nvidia-docker2
            sudo systemctl restart docker

            # depending on the configuration of your docker daemon, you may need to restart the Kasm agent container
            sudo docker restart kasm_agent
```

## GPU Graphics Acceleration

Graphics acceleration is supported on any Kasm maintained Workspace that bases from the
**kasmweb/core-ubuntu-focal** version 1.11.0 or higher. This includes many single app Workspaces and the
full Ubuntu Focal desktop **kasmweb/ubuntu-focal-desktop**. Building custom Workspaces that have GPU acceleration
is also possible if you base from **kasmweb/core-ubuntu-focal**. In most cases, GPU acceleration will work with
all applications, without anything special in the derivative Workspace. In some cases, you may need to launch your
application with vglrun. Here is an example of a script that executes Google Chrome with vglrun. The IF statement
ensures the GPU has indeed been passed through, and if not launches Chrome normally. This is not necessary for
most applications, Chrome happens to be one application that this is necessary for.

```Bash
if [ -f /opt/VirtualGL/bin/vglrun ] && [ ! -z "\${KASM_EGL_CARD}" ] && [ ! -z "\${KASM_RENDERD}" ] && [ -O "\${KASM_RENDERD}" ] && [ -O "\${KASM_EGL_CARD}" ] ; then
    echo "Starting Chrome with GPU Acceleration on EGL device \${KASM_EGL_CARD}"
    vglrun -d "\${KASM_EGL_CARD}" /opt/google/chrome/google-chrome ${CHROME_ARGS} "\$@"
else
    echo "Starting Chrome"
    /opt/google/chrome/google-chrome ${CHROME_ARGS} "\$@"
fi
```

GPU acceleration can be verified for the desktop by running `glxinfo -B` in a terminal.

```{figure} /images/gpus/glxinfo.png
:align: center
**Verifying GPU Graphics Acceleration using glxinfo**
```

GPU accelerated can be verified in Chrome and Chromium based browsers by navigating to chrome://gpu

```{figure} /images/gpus/chrome_gpu.png
:align: center
**Verifying GPU Graphics Acceleration in Chrome**
```

```{note}
For graphics acceleration, only a single GPU can be used. For non-graphics uses of GPUs, multiple GPUs can
be passed through and used by the container. For multi-GPU systems, Kasm automatically load balances the
session containers across the available GPUs.
```

## Confirm Kasm GPUs

After installing Workspaces, the GPU driver, and GPU container toolkit; confirm Kasm is picking up the GPUs. Log into Kasm as an administrator, navigate to the Agents view under the Admin section. Here you will see a list of agents in the cluster, since this is a single server installation, there will only be one agent listed. Confirm that the agent shows 1 or more GPUs. If the agent lists 0 GPUs, see the [troubleshooting section](../guide/troubleshooting/gpu_issues.md).

```{figure} /images/gpus/agents_view.png
:align: center
**Agents Listing**
```

```{figure} /images/gpus/agent_details.png
:align: center
**Agent GPU Details**
```

## Resource Management

GPUs are treated similar to CPU cores within Kasm Workspaces. Each agent reports the number of CPUs, RAM, and
GPUs available to share between containers. By default, a Workspace set to 1 require 1 GPU will mean that an agent
with 1 GPU will only be able to support a single session of that workspace. Kasm can allow a single GPU to be shared
between multiple containers, this is known as overriding, see the
{ref}`compute-overrides` for more details. Overriding the agent to have
4 GPUs would allow up to 4 containers with the GPU passed through, if each were assigned a single GPU.
In the context of graphics acceleration, only a single GPU can be passed through. Only non-graphics applications
support multiple GPUs per container to be used.

## Modify Workspace

In order for sessions to utilize a GPU, the Workspace must be set to require a GPU. Login as administrator, navigate to Workspaces,
in the Workspace list find the target Workspace. Expand the menu by clicking the arrow icon next to the Workspace you want to have a
GPU and click Edit. Change the GPU count to the desired number.

```{figure} /images/images/image_resources.webp
:align: center
**Create ML Desktop session**
```

## Building ML Desktop (Optional)

Kasm maintains an open source project, [Workspaces ML Desktop](https://github.com/kasmtech/workspaces-machine-learning),
a desktop environment intended for developers working on machine learning. This Workspace contains common development tools and
frameworks for ML development, but can be used in general for development. The following shows how to download and build
the ML Workspace.

```Bash
git clone https://github.com/kasmtech/workspaces-machine-learning.git
cd workspaces-machine-learning
sudo docker build -t kasmweb/desktop-ml:latest .
```

```{note}
If you build images directly on a Kasm Workspaces server, you must disable automatic image pruning in the {ref}`agent-settings`. While building locally is fine for testing, the build is better done in a CI pipeline and pushed to a central docker registry. Workspaces can be configured to automatically pull image updates from a central Docker registry.
```

The final step is to define a Workspace that utilizes this. Log into Kasm Workspaces as an administrator and navigate to Workspaces in the Admin panel. Click the Add Workspace button. Fill out the information as shown in the following screenshot. Adjust the cores, memory, and GPU settings as desired. Scroll down and enter the categories Development and Desktop in the Categories field, this ensures the Workspace shows up in those categories on the Workspaces home page.

```{figure} /images/images/image_create.webp
:align: center
**Create new Workspace**
```

There are many options and settings that can be applied, such as persistent profiles or two-factor authentication. See our various how-to and configuration guides in this documentation for more details.