---
myst:
  html_meta:
    "description lang=en": "Manual and automated upgrade process documentation for Kasm Workspaces multi server deployments."
    "keywords": "Kasm, Workspace, Multi-server, Multi, Server, Upgrade"
    "property=og:locale": "en_US"
---
```{title} Multi Server Upgrade
```

# Multi Server Upgrade

```{warning}
When running any upgrade please ensure there are no active Kasm Workspaces sessions running. On large deployments this will likely require a maintenance window.
```

```{note}
While not required, making a backup using a VM snapshot or other method of the servers in the installation is recommended as a precaution to provide a recovery point if something goes wrong.
```

## Manually

Please read through the entire process before getting started.

Issues can be reported on the [Kasm Workspaces Issues Page](https://github.com/kasmtech/workspaces-issues/issues)

**Component Registration Token**

The **Component Registration Token** is used during the upgrade process to register new components that were not present on earlier versions of Kasm Workspaces.

Login to the Workspaces UI as an administrator. Retrieve the value of  **Component Registration Token** from the Global Settings.

```{include} image_warning.md
```

- Stop all Kasm Services on all Servers

```Bash
sudo /opt/kasm/bin/stop
```

### Upgrade Database Server Role

Create a Database Backup

- Ensure Kasm services are stopped on all hosts. Database corruption can occur if Kasm services are still connected to the database when it goes down.

```{note}
  If the database is a standalone remote database refer here for backup instructions. [](../how_to/remote_database.md#backing-up-the-postgresql-server).
```

```Bash
sudo /opt/kasm/bin/stop
```

- Execute the database backup utility

```{parsed-literal}
sudo mkdir -p /opt/kasm/backups/
sudo bash /opt/kasm/{{ previous_release }}/bin/utils/db_backup -f /opt/kasm/backups/kasm_db_backup.tar -p /opt/kasm/{{ previous_release }}/
```

- Verify the presence and location of the database backup

```Bash
sudo ls -al /opt/kasm/backups/kasm_db_backup.tar
```

- Download and extract the new installation media

```{parsed-literal}
cd /tmp/
curl -O {{ release_url }}
tar -xf kasm_release_*.tar.gz
```

```{note}
When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see {doc}`System Requirements <../install/system_requirements>` for details.
```

- Get the existing database password for use in the subsequent commands.

```{parsed-literal}
sudo grep " password" /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
```

- Get the existing redis password for use in the subsequent commands.

```{parsed-literal}
sudo grep "redis_password" /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
```

**Perform a clean install**

Perform a clean install of the Database Role Services

```{note}
If the database is a standalone remote database refer here for clean install instructions. [](../how_to/remote_database.md#initializing-postgresql-server).
```

- Install kasm from the release media downloaded in the prior steps.

  - When performing an offline update add these flags `--offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh -S db -D -Q <DATABASE_PASSWORD> -R <REDIS_PASSWORD>
```

### Restore and upgrade the database

Restore and update the database from the prior version

- Ensure all Kasm services are stopped on all hosts

```{note}
If the database is a standalone remote database refer here for restoration instructions. [](../how_to/remote_database.md#restoring-the-postgresql-server-from-a-backup).
```

```Bash
sudo /opt/kasm/bin/stop
```

- Execute the database restore command

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_restore -f /opt/kasm/backups/kasm_db_backup.tar -p  /opt/kasm/{{ release }}/
```

- Perform an upgrade of the database schema

```{note}
If the database is a standalone remote database refer here for upgrade instructions. [](../how_to/remote_database.md).
```

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_upgrade -p /opt/kasm/{{ release }}
```

- **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

```{parsed-literal}
diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml  /opt/kasm/{{ release }}/docker/docker-compose.yaml
```

- (Optional) Install the Kasm {{ release }} default images for the platform

  - With Kasm Workspaces 1.13.0 and newer the Kasm team recommends using the Workspaces Registry to install Workspaces rather than seeding the images during the installation.
  - Seeding workspaces is still needed when doing an offline Kasm installation as the official Kasm Workspaces Registry will be unavailable.
  - If doing an offline upgrade first extract the default image seed data from the workspace images tar.

    For x86 (amd64) platforms:

    ```{parsed-literal}
    tar xf <KASM_WORKSPACE_IMAGES_TAR> --strip-components=1 -C /opt/kasm/{{ release }}/conf/database/seed_data/ workspace_images/default_images_amd64.yaml
    ```

    For ARM (arm64) platforms:

    ```{parsed-literal}
    tar xf <KASM_WORKSPACE_IMAGES_TAR> --strip-components=1 -C /opt/kasm/{{ release }}/conf/database/seed_data/ workspace_images/default_images_arm64.yaml
    ```

For x86 (amd64) platforms:

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_init -s /opt/kasm/{{ release }}/conf/database/seed_data/default_images_amd64.yaml
```

For ARM (arm64) platforms:

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_init -s /opt/kasm/{{ release }}/conf/database/seed_data/default_images_arm64.yaml
```

- Start the Kasm services

```Bash
sudo /opt/kasm/bin/start
```

### Upgrade Web App Server Role

- Stop existing Kasm Services

```Bash
sudo /opt/kasm/bin/stop
```

- Download and extract the new installation media

```Bash
cd /tmp/
curl -O |release_url|
tar -xf kasm_release_*.tar.gz
```

```{note}
When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see {doc}`System Requirements <../install/system_requirements>` for details.
```

- Install kasm from the release media downloaded in the prior steps.

  - When performing an offline update add this flag `--offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh -S app -D -q <DATABASE_HOSTNAME> -Q <DATABASE_PASSWORD> -R [REDIS_PASSWORD]
```

- Copy **manager_id** and **server_hostname** from the old configuration into the new

```{parsed-literal}
grep manager_id /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
grep server_hostname /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
sudo vi /opt/kasm/{{ release }}/conf/app/api.app.config.yaml
```

- **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

```{parsed-literal}
diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml  /opt/kasm/{{ release }}/docker/docker-compose.yaml
```

- Start the Kasm services

```Bash
sudo /opt/kasm/bin/start
```

### Upgrade Agent Server Role

- Ensure all Kasm services are stopped

```Bash
sudo /opt/kasm/bin/stop
```

- Download and extract the new installation media

```{parsed-literal}
cd /tmp/
curl -O {{ release_url }}
tar -xf kasm_release_*.tar.gz
```

- Get the existing manager token for use in the subsequent commands.

```{parsed-literal}
sudo grep "token" /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml
```

```{note}
When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see {doc}`System Requirements <../install/system_requirements>` for details.
```

- Install kasm from the release media downloaded in the prior steps.

  - When performing an offline update add these flags `--offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR> --offline-plugin <KASM_PLUGIN_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh -S agent -D -p <AGENT_HOSTNAME> -m <MANAGER_HOSTNAME> -M <MANAGER_TOKEN>
```

- Copy the server_id and the public_hostname properties from the old agent to the new

```{parsed-literal}
grep server_id /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml
grep public_hostname /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml
sudo vi /opt/kasm/{{ release }}/conf/app/agent.app.config.yaml
```

- Copy the auto-generated nginx configs for any sessions that may exist on the Agent

```{parsed-literal}
sudo cp /opt/kasm/{{ previous_release }}/conf/nginx/containers.d/* /opt/kasm/{{ release }}/conf/nginx/containers.d/
```

- **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

```{parsed-literal}
diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml  /opt/kasm/{{ release }}/docker/docker-compose.yaml
diff /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml  /opt/kasm/{{ release }}/conf/app/agent.app.config.yaml
```

- Start the Kasm services

```Bash
sudo /opt/kasm/bin/start
```

- Verify the Agent Service is properly checking into the Manager Service. Log into the UI as an Administrator.
  Select **Agents** and verify the Agent is listed with a **Last Reported** time of less than 1 minute.
  If the agent has not checked in, it likely can’t resolve or connect to \[AGENT_HOSTNAME\]:443 . Inspect the logs for details.

```bash
sudo tail -f /opt/kasm/current/log/agent.log
```

### (Optional) Install the Connection Proxy (Guac/rdp-gateway) Role(s)

The Connection Proxy role is new in Workspaces 1.12.0. This service is used to connect to VM/Hardware running RDP, VNC, or SSH. 
If these capabilities are not needed, this role does not need to be installed.

Login to the Workspaces UI as an administrator. Retrieve the value of  **Component Registration Token** from the Global
Settings.


- Download and extract the new installation media

```{parsed-literal}
cd /tmp/
curl -O {{ release_url }}
tar -xf kasm_release_*.tar.gz
```

- Install the Connection Proxy role from the release media downloaded in the prior steps.

  - When performing an offline update add these flags `--offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh --role guac --api-hostname [MANAGER_HOSTNAME] --public-hostname [CONNECTION_PROXY_HOSTNAME] --registration-token [SERVICE_REGISTRATION_TOKEN] -D
```

- Copy the id and the token properties from the old connection_proxy to the new
  
  - The *token* field of *kasmguac.app.config.yaml* has been renamed to *auth_token* in the Kasm Workspaces 1.13.0 release 

```{parsed-literal}
grep id /opt/kasm/{{ previous_release }}/conf/app/kasmguac.app.config.yaml
/opt/kasm/bin/utils/yq_$(uname -m) -r '.api.auth_token' /opt/kasm/{{ previous_release }}/conf/app/kasmguac.app.config.yaml
sudo vi /opt/kasm/{{ release }}/conf/app/kasmguac.app.config.yaml
```

In Kasm Workspaces version 1.16.0 The Kasm RDP Gateway service was added, when upgrading from any Kasm Workspaces version <= 1.15.0 to any Kasm Workspaces version >= 1.16.0 please perform the below steps to register the new services:

- Set the registration_token for the rdp_gateway using the **Component Registration Token** retrieved from the UI earlier.
  
  - Replace `<registration_token>` below with the  **Component Registration Token**

```{parsed-literal}
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.rdp-gateway.registration_token = "<registration_token>"' /opt/kasm/{{ release }}/conf/app/passthrough.app.config.yaml
```

- Set the registration_token for the rdp_https_gateway using the **Component Registration Token** retrieved from the UI earlier.
  
  - Replace `<registration_token>` below with the  **Component Registration Token**

```{parsed-literal}
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.rdp-gateway.registration_token = "<registration_token>"' /opt/kasm/{{ release }}/conf/app/rdp_https_gateway.app.config.yaml
```

```Bash
sudo /opt/kasm/bin/start
```

## Upgrade Script

Since Kasm Workspaces version 1.11.0 an upgrade script is included with the installation package under kasm_release/upgrade.sh.

````{note}
Before starting the upgrade process please ensure that the Web App servers are stopped. If left running there is a chance for database corruption on the Webb App servers
```Bash 
sudo /opt/kasm/bin/stop
```
````

The upgrade script can be found in the installation package under kasm_release/upgrade.sh.

These instructions assume there are 3 servers with the app, db, and agent roles.

Login to the db server and execute:

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role db
```

````

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
curl -O {{ amd_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role db --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
curl -O {{ arm_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role db --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

Login to the agent server and execute:

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role agent
```

````

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
curl -O {{ amd_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role agent --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
curl -O {{ arm_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role agent --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

Login to the Web App server and execute:

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role app
```

```` 

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role app --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role app --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

Login to the connection proxy server and execute:

```{note}
The **Component Registration Token** can be obtained by viewing its field in the Kasm Workspaces Deployment's [Global Server Settings](../guide/settings.md#authentication)
```

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role guac --registration-token [COMPONENT_REGISTRATION_TOKEN]
```

````

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role guac --registration-token [COMPONENT_REGISTRATION_TOKEN] --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role guac --registration-token [COMPONENT_REGISTRATION_TOKEN] --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

Here are the available helper flags for the upgrade.sh script:

```text
Flag                                        Description
---------------------------------------------------------------------------------------------------------------
| -h|--help                     | Display this help menu                                                      |
| -L|--proxy-port               | Default Proxy Listening Port                                                |
| -s|--offline-service          | Path to the tar.gz service images offline installer                         |
| -x|--offline-plugin           | Path to the tar.gz plugin images offline installer                          |
| -S|--role                     | Role to Upgrade: [app|db|agent|remote_db|guac|proxy]                        |
| -p|--public-hostname          | Agent/Component <IP/Hostname> used to register with deployment.             |
| -g|--database-master-user     | Database master username required for remote DB                             |
| -G|--database-master-password | Database master password required for remote DB                             |
| -q|--db-hostname              | Database Hostname needed when upgrading agent and pulling images            |
| -T|--db-port                  | Database port needed when upgrading agent and pulling images (default 5432) |
| -Q|--db-password              | Database Password needed when upgrading agent and pulling images            |
| -b|--no-check-disk            | Do not check disk space                                                     |
| -n|--api-hostname             | Set API server hostname                                                     |
| -A|--enable-lossless          | Enable lossless streaming option (1.12 and above)                           |
| -O|--use-rolling-images       | Use rolling Service images                                                  |
| -k|--registration-token       | Register a component with an existing deployment.                           |
| --skip-egress                 | Do not install egress network plugin or dependancies                        |
---------------------------------------------------------------------------------------------------------------
```

### Automatic Upgrade Troubleshooting

The upgrade.sh script creates a log file as it runs, this file is removed upon completion of a successful upgrade. However, if something does go wrong the logfile will be available from the directory the upgrade.sh script was executed from in the format *kasm_upgrade_${TIMESTAMP}.log*. This file will be important for diagnosing the error that caused the upgrade to fail and will be requested when submitting a support ticket with Kasm Technologies.

```{include} upgrade_images.md
```