---
myst:
  html_meta:
    "description lang=en": "Learn how to configure autoscaling for Kasm Workspaces on Proxmox. This guide covers prerequisites, Proxmox user permissions, VM provider setup, autoscaling testing, and optimization to ensure dynamic resource allocation."
    "keywords": "Kasm, Proxmox, Pooling, Pool, autoscale, autoscaling, auto, scale, VM"
    "property=og:locale": "en_US"
---
```{title} Proxmox AutoScale
```

# Proxmox AutoScale

```{contents} Table of Contents
:depth: 3
:local:
```

This guide will walk you through configuring autoscaling for Kasm Workspaces on Proxmox. Autoscaling in Kasm Workspaces automatically provisions and destroys agents based on user demand, ensuring optimized resource utilization and cost efficiency.


```{raw} html
<iframe src="https://www.youtube.com/embed/nXIBGs_WJcs?si=XG890-8eURPtGP5s" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
```

## Prerequisites

* Access to Proxmox: Ensure you have admin access to your Proxmox environment
* Kasm Workspaces Installed: A basic setup of Kasm Workspaces must already exist
* Understand Key Concepts:
  * **Zones**: Logical groupings of Kasm services for geographical or organizational segmentation
  * **Pools**: Logical groupings of Kasm Docker Agents and Server Pools for load balancing
* Plan Your Configuration:
  * Understand your deployment zone requirements
  * Have access to your Proxmox server details; target clusters, resource pool, storage, networks, etc.
  * Create template images in the resource pool you have designated for your autoscale agents, must include the qemu guest tools


## Setup your Proxmox environment


### Create Proxmox Pool

In your Proxmox dashboard, go to "Permissions" -> "Pools" -> "Create" to create a new `kasm-autoscale` pool.
```{figure} /images/autoscaling/providers/proxmox/proxmox_create_pool.png
:align: center
**Create a pool in Proxmox**
```

### Create Proxmox user

Go to "Permissions" -> "Users" -> "Add" to create a new user. You don't have to create a new account for autoscaling but we strongly recommend it.
```{figure} /images/autoscaling/providers/proxmox/proxmox_create_user.png
:align: center
**Create a user in Proxmox**
```

### Create API Token

Go to "Permissions" -> "API Tokens" -> "Add" to create a new Proxmox API token to use with Kasm. If you're using an existing Proxmox account, we recommend you leave "Privilege Separation" enabled. If you're using a dedicated Proxmox account, you can disable "Privilege Separation". Make sure you save your generated Token ID and Secret securely as you cannot see them again.
```{figure} /images/autoscaling/providers/proxmox/proxmox_create_api_token.png
:align: center
**Create an API token in Proxmox**
```

### Create Role

To create a Role, go to "Permissions" -> "Roles" -> "Create". You need the following privileges for Kasm autoscaling to function properly:
  * Pool.Audit
  * VM.Allocate
  * Datastore.AllocateSpace
  * SDN.Use
  * VM.Audit
  * VM.Clone
  * VM.Config.CDROM
  * VM.Config.CPU
  * VM.Config.Disk
  * VM.Config.HWType
  * VM.Config.Memory
  * VM.Config.Network
  * VM.Config.Options
  * VM.Monitor
  * VM.PowerMgmt 

```{figure} /images/autoscaling/providers/proxmox/proxmox_create_role.png
:align: center
**Create a Role in Proxmox**
```

### Assign Permissions

Finally, assign appropriate permissions to your user by going to "Permissions" -> "Add" -> "User Permission". Select the appropriate user and role, and assign the following permissions
  * /sdn/zones/\<networkzone\>
  * /storage/\<storagepool\>
  * /pool/KasmPool

```{figure} /images/autoscaling/providers/proxmox/proxmox_create_permissions.png
:align: center
**Assign permissions in Proxmox**
```

````{note}
If you are using an existing account and have privilege separation enabled, you need to set these permissions from "Permissions" -> "Add" -> "API Token Permission"
````

### Create a VM template

```{eval-rst}
Create the appropriate VM template based on whether you are implementing Server AutoScaling or Docker AutoScaling on Proxmox.
  * For Windows AutoScaling, follow the :ref:`Windows Templating Guide<proxmox-windows-templating>`
  * For Docker Agent AutoScaling, follow the :ref:`Linux Templating Guide<proxmox-linux-templating>`
```

(proxmox-linux-templating)=
#### Linux Templating

- Go to your cluster -> local storage (e.g local (pve)) -> "Upload"
- Upload your Linux installation ISO file (you can download Ubuntu Server from [here](https://ubuntu.com/download/server))
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_upload_ubuntu_iso.png
  :align: center
  **Upload Linux ISO file to Proxmox**
  ```
- Click "Create VM" from the top right corner
  * General Settings
    * Name: Give your template a name (e.g `ubuntu-server-template`)
    * Resource Pool: Set it to the resource pool you created earlier
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm1.png
    :align: center
    **Create Proxmox Linux VM: General Settings**
    ```
  * OS Settings
    * Select `Use CD/DVD disc image file (iso)`
    * Storage: Select your local storage to which you uploaded your ISO file
    * ISO Image: Select the Linux ISO image you uploaded
    * Guest OS Type: Set to Linux
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm2.png
    :align: center
    **Create Proxmox Linux VM: OS Settings**
    ```
  * System Settings
    * Machine: Select `q35`
    * SCSI controller: Select `VirtIO SCSI single`
    * Enable `Qemu Agent`
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm3.png
    :align: center
    **Create Proxmox Linux VM: System Settings**
    ```
  * Disk Settings
    * Bus/Device: Select `VirtIO Block`
    * Disk Size: Choose the amount of size you want to allocate to your VM (e.g 45GB)
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm4.png
    :align: center
    **Create Proxmox Linux VM: Disk Settings**
    ```
  * CPU Settings
    * Cores: Set to the number of cores you want to allocate to your VM (e.g 4)
    * Type: Select `host`
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm5.png
    :align: center
    **Create Proxmox Linux VM: CPU Settings**
    ```
  * Memory Settings
    * Memory: Set to the amount of memory you want to allocate to your VM in MiB (e.g 4096)
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm6.png
    :align: center
    **Create Proxmox Linux VM: Memory Settings**
    ```
  * Network Settings
    * Model: Select `VirtIO (paravirtualized)`
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_linux_vm7.png
    :align: center
    **Create Proxmox Linux VM: Network Settings**
    ```
  * Confirm your settings and click "Finish"
- Right-click your created VM and select "Start". Then double-click to open the noVNC window.
- Follow the on-screen instructions to install Ubuntu Server
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_ubuntu_install_screen1.png
  :align: center
  **Install Ubuntu on Proxmox**
  ```
- Select "Install OpenSSH server" to be able to SSH into your autoscaled agents.
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_ubuntu_install_screen2.png
  :align: center
  **Select "Install OpenSSH server"**
  ```
- Once the installation is finished, click "Reboot Now" 
- Go back to your VM settings on Proxmox -> "Hardware" -> Double-click the `CD/DVD Drive` and select "Do not use any media" to remove the Linux installation media.
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_ubuntu_remove_iso.png
  :align: center
  **Remove Ubuntu ISO from your VM"**
  ```
- You can now SSH into your VM with your credentials instead of using noVNC for ease-of-access
- Install and enable `qemu-guest-agent` on your VM
  ```
  sudo apt update
  sudo apt install qemu-guest-agent -y
  sudo systemctl start qemu-guest-agent
  sudo systemctl enable qemu-guest-agent
  sudo reboot
  ```
- Reset your Machine IDs to make your VM suitable for templating
  ```
  sudo truncate -s 0 /etc/machine-id
  sudo truncate -s 0 /var/lib/dbus/machine-id
  ```
- Optionally, you can pre-load workspace images on your autoscaled agent VMs so that workspaces launch instantly after provisioning, without waiting for Kasm to pull the necessary Docker images. Read the [Pre-load Workspace Images on Agents guide](../preload_images_on_agents.md) to learn more
- Shutdown your VM
- Once the VM is powered off, right-click and select "Convert to template"
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_ubuntu_create_template.png
  :align: center
  **Create VM template in Proxmox"**
  ```

(proxmox-windows-templating)=
#### Windows Templating

- Go to your cluster -> local storage (e.g local (pve)) -> "Upload"
- Upload your Windows installation ISO file (you can download Windows Server 2022 from [here](https://www.microsoft.com/en-us/evalcenter/download-windows-server-2022))
- Also upload the VirtIO drivers ISO file (you can download VirtIO tools from [here](https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.266-1/))
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_upload_windows_iso.png
  :align: center
  **Upload Windows ISO and VirtIO ISO to Proxmox**
  ```
- Click "Create VM" from the top right corner
  * General Settings
    * Name: Give your template a name (e.g `windows-server-2022-template`)
    * Resource Pool: Set it to the resource pool you created earlier
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm1.png
    :align: center
    **Create Proxmox Windows VM: General Settings**
    ```
  * OS Settings
    * Select `Use CD/DVD disc image file (iso)`
    * Storage: Select your local storage to which you uploaded your ISO file
    * ISO Image: Select the Windows Server ISO image you uploaded
    * Guest OS Type: Set to `Microsoft Windows`
    * Select `Add additional drive for VirtIO drivers`
    * ISO Image: Select the VirtIO ISO image you uploaded
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm2.png
    :align: center
    **Create Proxmox Windows VM: OS Settings**
    ```
  * System Settings
    * Machine: Select `q35`
    * SCSI controller: Select `VirtIO SCSI single`
    * Enable `Qemu Agent`
    * Disable `Add EFI Disk`
    * Disable `Add TPM`
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm3.png
    :align: center
    **Create Proxmox Windows VM: System Settings**
    ```
  * Disk Settings
    * Bus/Device: Select `SCSI`
    * Cache: Select `Write back`
    * Disk Size: Choose the amount of size you want to allocate to your VM (e.g 45GB)
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm4.png
    :align: center
    **Create Proxmox Windows VM: Disk Settings**
    ```
  * CPU Settings
    * Cores: Set to the number of cores you want to allocate to your VM (e.g 4)
    * Type: Select `host`
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm5.png
    :align: center
    **Create Proxmox Windows VM: CPU Settings**
    ```
  * Memory Settings
    * Memory: Set to the amount of memory you want to allocate to your VM in MiB (e.g 4096)
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm6.png
    :align: center
    **Create Proxmox Windows VM: Memory Settings**
    ```
  * Network Settings
    * Model: Select `VirtIO (paravirtualized)`
    * Click "Next"
    ```{figure} /images/autoscaling/providers/proxmox/proxmox_create_windows_vm7.png
    :align: center
    **Create Proxmox Windows VM: Network Settings**
    ```
  * Confirm your settings and click "Finish"
- Right-click your created VM and select "Start". Then, double-click to open the noVNC window.
- Press any key to boot from the attached Windows Installation media
- You will now see the Windows Installation screen. Proceed with the Installation.
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_windows_install_screen1.png
  :align: center
  **Follow on-screen instructions to Install Windows**
  ```
- By default, you may not be able to see the list of available disks to install Windows. To fix this, you need to install the VirtIO SCSI controller. Click "Load driver" to list all the compatible VirtIO SCSI controllers and install the appropriate driver.
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_windows_install_screen2.png
  :align: center
  **Install the appropriate VirtIO SCSI contoller**
  ```
- The available disks must now be listed and you can choose the Windows boot disk you created to install Windows. 
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_windows_install_screen3.png
  :align: center
  **Install Windows**
  ```
- Once Windows is installed, you need to install the other VirtIO drivers like the Network driver. Simply navigate to your VirtIO disk from your File Explorer and install the drivers by running the installer.
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_windows_install_screen4.png
  :align: center
  **Install VirtIO drivers**
  ```
- You also need to install the QEMU Guest Agent tools. These can be found in the same VirtIO drive in the `guest-agent` folder.
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_windows_install_screen5.png
  :align: center
  **Install QEMU Guest Agent**
  ```
- Reboot your VM
- Open a PowerShell window as administrator and set the `ExecutionPolicy` to `Unrestricted` so that Kasm can run the startup script when a VM is provisioned
  ```
  Set-ExecutionPolicy Unrestricted
  ```
- Search for "Remote Desktop Settings" and enable "Remote Desktop" on your VM
  ```{figure} /images/autoscaling/providers/proxmox/proxmox_windows_install_screen6.png
  :align: center
  **Enable Remote Desktop**
  ```
- Now, you can install any custom software that you'd like to have on your VM (e.g Microsoft Office)
- Shutdown your VM
- Once the VM is powered off, right-click on it and select "Convert to template"
- Go to your created template -> "Hardware" -> Remove the attached Windows ISO and VirtIO ISO disks

For more details about Windows Templating, refer to the [Windows AutoScale Template Creation Guide](../windows_autoscale_templates.md)

## Configure VM Provider Details on Kasm

```{eval-rst}
* Follow :ref:`autoscale_docker_config` or :ref:`autoscale_server_config` to create to create a new AutoScale config, or select **Create New** in **VM Provider Configs** if you already have one.
* Set Provider to Proxmox
* Configure the following settings:
```

```{include} /guide/compute/vm_providers/proxmox.md
```

* Submit the Provider Config

## Test your Proxmox Autoscaling setup

If you have configured non-zero Standby/Minimum Available Session values agents should start provisioning immediately. Otherwise, try launching multiple workspaces to increase resource utilization, prompting Kasm to autoscale new agents.
* Provision a Workspace
  * Go to Workspaces > Registry
  * Make multiple workspaces available
* Go to the Workspaces dashboard and launch sufficient workspace sessions to exceed your resource standby thresholds
* Monitor the provisioning of new agents by going to "Infrastructure" -> "Agents"
* Verify new VM instances in Proxmox
* Check Downscaling
  * Terminate sessions to reduce resource usage
  * Confirm that Kasm removes agents after the back-off period