Multi Server Upgrade

Warning

When running any upgrade please ensure there are no active Kasm Workspaces sessions running. On large deployments this will likely require a maintenance window.

Note

While not required, making a backup using a VM snapshot or other method of the servers in the installation is recommended as a precaution to provide a recovery point if something goes wrong.

Manually

Please read through the entire process before getting started.

Issues can be reported on the Kasm Workspaces Issues Page

Component Registration Token

The Component Registration Token is used during the upgrade process to register new components that were not present on earlier versions of Kasm Workspaces.

Login to the Workspaces UI as an administrator. Retrieve the value of Component Registration Token from the Global Settings.

Warning

The Kasm Workspaces 1.16.1 upgrade will optionally install a new set of default desktop and application images. Please ensure 50GB of free space is available on the Kasm server ( or Agent role server for multi-server installations) prior to upgrading if selecting the options to seed new images.

• Stop all Kasm Services on all Servers

sudo /opt/kasm/bin/stop

Upgrade Database Server Role

Create a Database Backup

• Ensure Kasm services are stopped on all hosts. Database corruption can occur if Kasm services are still connected to the database when it goes down.

Note

If the database is a standalone remote database refer here for backup instructions. Backing up the PostgreSQL Server.

sudo /opt/kasm/bin/stop

• Execute the database backup utility

sudo mkdir -p /opt/kasm/backups/
sudo bash /opt/kasm/1.16.0/bin/utils/db_backup -f /opt/kasm/backups/kasm_db_backup.tar -p /opt/kasm/1.16.0/

• Verify the presence and location of the database backup

sudo ls -al /opt/kasm/backups/kasm_db_backup.tar

• Download and extract the new installation media

cd /tmp/
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_*.tar.gz

Note

When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see System Requirements for details.

• Get the existing database password for use in the subsequent commands.

sudo grep " password" /opt/kasm/1.16.0/conf/app/api.app.config.yaml

• Get the existing redis password for use in the subsequent commands.

sudo grep "redis_password" /opt/kasm/1.16.0/conf/app/api.app.config.yaml

Perform a clean install

Perform a clean install of the Database Role Services

Note

If the database is a standalone remote database refer here for clean install instructions. Initializing PostgreSQL Server.

• Install kasm from the release media downloaded in the prior steps.

  • When performing an offline update add these flags --offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>

sudo bash kasm_release/install.sh -S db -D -Q <DATABASE_PASSWORD> -R <REDIS_PASSWORD>

Restore and upgrade the database

Restore and update the database from the prior version

• Ensure all Kasm services are stopped on all hosts

Note

If the database is a standalone remote database refer here for restoration instructions. Restoring the PostgreSQL server from a backup.

sudo /opt/kasm/bin/stop

• Execute the database restore command

sudo /opt/kasm/1.16.1/bin/utils/db_restore -f /opt/kasm/backups/kasm_db_backup.tar -p  /opt/kasm/1.16.1/

• Perform an upgrade of the database schema

Note

If the database is a standalone remote database refer here for upgrade instructions. Standalone Remote Database.

sudo /opt/kasm/1.16.1/bin/utils/db_upgrade -p /opt/kasm/1.16.1

• Advanced: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

diff /opt/kasm/1.16.0/docker/docker-compose.yaml  /opt/kasm/1.16.1/docker/docker-compose.yaml

• (Optional) Install the Kasm 1.16.1 default images for the platform

  • With Kasm Workspaces 1.13.0 and newer the Kasm team recommends using the Workspaces Registry to install Workspaces rather than seeding the images during the installation.

  • Seeding workspaces is still needed when doing an offline Kasm installation as the official Kasm Workspaces Registry will be unavailable.

  • If doing an offline upgrade first extract the default image seed data from the workspace images tar.

    For x86 (amd64) platforms:

    tar xf <KASM_WORKSPACE_IMAGES_TAR> --strip-components=1 -C /opt/kasm/1.16.1/conf/database/seed_data/ workspace_images/default_images_amd64.yaml

    For ARM (arm64) platforms:

    tar xf <KASM_WORKSPACE_IMAGES_TAR> --strip-components=1 -C /opt/kasm/1.16.1/conf/database/seed_data/ workspace_images/default_images_arm64.yaml

For x86 (amd64) platforms:

sudo /opt/kasm/1.16.1/bin/utils/db_init -s /opt/kasm/1.16.1/conf/database/seed_data/default_images_amd64.yaml

For ARM (arm64) platforms:

sudo /opt/kasm/1.16.1/bin/utils/db_init -s /opt/kasm/1.16.1/conf/database/seed_data/default_images_arm64.yaml

• Start the Kasm services

sudo /opt/kasm/bin/start

Upgrade Web App Server Role

• Stop existing Kasm Services

sudo /opt/kasm/bin/stop

• Download and extract the new installation media

cd /tmp/
curl -O |release_url|
tar -xf kasm_release_*.tar.gz

Note

When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see System Requirements for details.

• Install kasm from the release media downloaded in the prior steps.

  • When performing an offline update add this flag --offline-service <KASM_SERVICE_IMAGES_TAR>

sudo bash kasm_release/install.sh -S app -D -q <DATABASE_HOSTNAME> -Q <DATABASE_PASSWORD> -R [REDIS_PASSWORD]

• Copy manager_id and server_hostname from the old configuration into the new

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.server.server_hostname = load("/opt/kasm/1.16.0/conf/app/api.app.config.yaml").server.server_hostname' /opt/kasm/1.16.1/conf/app/api.app.config.yaml
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.manager.manager_id = load("/opt/kasm/1.16.0/conf/app/api.app.config.yaml").manager.manager_id' /opt/kasm/1.16.1/conf/app/api.app.config.yaml

• Advanced: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

diff /opt/kasm/1.16.0/docker/docker-compose.yaml  /opt/kasm/1.16.1/docker/docker-compose.yaml

• Start the Kasm services

sudo /opt/kasm/bin/start

Upgrade Agent Server Role

• Ensure all Kasm services are stopped

sudo /opt/kasm/bin/stop

• Download and extract the new installation media

cd /tmp/
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_*.tar.gz

• Get the existing manager token for use in the subsequent commands.

sudo grep "token" /opt/kasm/1.16.0/conf/app/agent.app.config.yaml

Note

When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see System Requirements for details.

• Install kasm from the release media downloaded in the prior steps.

  • When performing an offline update add these flags --offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR> --offline-plugin <KASM_PLUGIN_IMAGES_TAR>

sudo bash kasm_release/install.sh -S agent -D -p <AGENT_HOSTNAME> -m <MANAGER_HOSTNAME> -M <MANAGER_TOKEN>

• Copy the server_id and the public_hostname properties from the old agent to the new

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.agent.server_id = load("/opt/kasm/1.16.0/conf/app/agent.app.config.yaml").agent.server_id' /opt/kasm/1.16.1/conf/app/agent.app.config.yaml
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.agent.public_hostname = load("/opt/kasm/1.16.0/conf/app/agent.app.config.yaml").agent.public_hostname' /opt/kasm/1.16.1/conf/app/agent.app.config.yaml

• Copy the auto-generated nginx configs for any sessions that may exist on the Agent

sudo cp /opt/kasm/1.16.0/conf/nginx/containers.d/* /opt/kasm/1.16.1/conf/nginx/containers.d/

• Advanced: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

diff /opt/kasm/1.16.0/docker/docker-compose.yaml  /opt/kasm/1.16.1/docker/docker-compose.yaml
diff /opt/kasm/1.16.0/conf/app/agent.app.config.yaml  /opt/kasm/1.16.1/conf/app/agent.app.config.yaml

• Start the Kasm services

sudo /opt/kasm/bin/start

• Verify the Agent Service is properly checking into the Manager Service. Log into the UI as an Administrator. Select Agents and verify the Agent is listed with a Last Reported time of less than 1 minute. If the agent has not checked in, it likely can’t resolve or connect to [AGENT_HOSTNAME]:443 . Inspect the logs for details.

sudo tail -f /opt/kasm/current/log/agent.log

(Optional) Install the Connection Proxy (Guac/rdp-gateway) Role(s)

The Connection Proxy role is new in Workspaces 1.12.0. This service is used to connect to VM/Hardware running RDP, VNC, or SSH. If these capabilities are not needed, this role does not need to be installed.

Login to the Workspaces UI as an administrator. Retrieve the value of Component Registration Token from the Global Settings.

• Download and extract the new installation media

cd /tmp/
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_*.tar.gz

• Install the Connection Proxy role from the release media downloaded in the prior steps.

  • When performing an offline update add these flags --offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>

sudo bash kasm_release/install.sh --role guac --api-hostname [MANAGER_HOSTNAME] --public-hostname [CONNECTION_PROXY_HOSTNAME] --registration-token [SERVICE_REGISTRATION_TOKEN] -D

• Copy the id and the token properties from the old connection_proxy to the new

  • The token field of kasmguac.app.config.yaml has been renamed to auth_token in the Kasm Workspaces 1.13.0 release

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.kasmguac.id = load("/opt/kasm/1.16.0/conf/app/kasmguac.app.config.yaml").kasmguac.id' /opt/kasm/1.16.1/conf/app/kasmguac.app.config.yaml
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.api.auth_token = load("/opt/kasm/1.16.0/conf/app/kasmguac.app.config.yaml").api.auth_token' /opt/kasm/1.16.1/conf/app/kasmguac.app.config.yaml

• Copy the id and the auth_token properties from the old rdp_gateway to the new

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.api.auth_token = load("/opt/kasm/1.16.0/conf/app/passthrough.app.config.yaml").api.auth_token' /opt/kasm/1.16.1/conf/app/passthrough.app.config.yaml
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.rdp-gateway.id = load("/opt/kasm/1.16.0/conf/app/passthrough.app.config.yaml").rdp-gateway.id' /opt/kasm/1.16.1/conf/app/passthrough.app.config.yaml

• Copy the id and the auth_token properties from the old rdp_https_gateway to the new

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.api.auth_token = load("/opt/kasm/1.16.0/conf/app/rdp_https_gateway.app.config.yaml").api.auth_token' /opt/kasm/1.16.1/conf/app/rdp_https_gateway.app.config.yaml
sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.rdp-gateway.id = load("/opt/kasm/1.16.0/conf/app/rdp_https_gateway.app.config.yaml").rdp-gateway.id' /opt/kasm/1.16.1/conf/app/rdp_https_gateway.app.config.yaml

In Kasm Workspaces version 1.16.0 The Kasm RDP Gateway service was added, when upgrading from any Kasm Workspaces version <= 1.15.0 to any Kasm Workspaces version >= 1.16.0 please perform the below steps to register the new services:

• Set the registration_token for the rdp_gateway using the Component Registration Token retrieved from the UI earlier.

  • Replace <registration_token> below with the Component Registration Token

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.rdp-gateway.registration_token = "<registration_token>"' /opt/kasm/1.16.1/conf/app/passthrough.app.config.yaml

• Set the registration_token for the rdp_https_gateway using the Component Registration Token retrieved from the UI earlier.

  • Replace <registration_token> below with the Component Registration Token

sudo /opt/kasm/bin/utils/yq_$(uname -m) -i '.rdp-gateway.registration_token = "<registration_token>"' /opt/kasm/1.16.1/conf/app/rdp_https_gateway.app.config.yaml

sudo /opt/kasm/bin/start

Upgrade Script

Since Kasm Workspaces version 1.11.0 an upgrade script is included with the installation package under kasm_release/upgrade.sh.

Note

Before starting the upgrade process please ensure that the Web App servers are stopped. If left running there is a chance for database corruption on the Webb App servers

sudo /opt/kasm/bin/stop

The upgrade script can be found in the installation package under kasm_release/upgrade.sh.

These instructions assume there are 3 servers with the app, db, and agent roles.

Login to the db server and execute:

Standard UpgradeOffline Upgrade x86-64Offline Upgrade arm64

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_1.16.1.98d6fa.tar.gz
sudo bash kasm_release/upgrade.sh --role db

Login to the agent server and execute:

Standard UpgradeOffline Upgrade x86-64Offline Upgrade arm64

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_1.16.1.98d6fa.tar.gz
sudo bash kasm_release/upgrade.sh --role agent

Login to the Web App server and execute:

Standard UpgradeOffline Upgrade x86-64Offline Upgrade arm64

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_1.16.1.98d6fa.tar.gz
sudo bash kasm_release/upgrade.sh --role app

Login to the connection proxy server and execute:

Note

The Component Registration Token can be obtained by viewing its field in the Kasm Workspaces Deployment’s Global Server Settings

Standard UpgradeOffline Upgrade x86-64Offline Upgrade arm64

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_1.16.1.98d6fa.tar.gz
sudo bash kasm_release/upgrade.sh --role guac --registration-token [COMPONENT_REGISTRATION_TOKEN]

Login to the dedicated proxy server and execute:

In this example we assume the proxy domain is proxy.example.com and the Web App hosting the API server is located at workspaces.example.com.

Standard UpgradeOffline Upgrade x86-64Offline Upgrade arm64

cd /tmp
curl -O https://kasm-static-content.s3.amazonaws.com/kasm_release_1.16.1.98d6fa.tar.gz
tar -xf kasm_release_1.16.1.98d6fa.tar.gz
sudo bash kasm_release/upgrade.sh --role proxy --api-hostname workspaces.kasmweb.com

Here are the available helper flags for the upgrade.sh script:

Flag                                        Description
---------------------------------------------------------------------------------------------------------------
| -h|--help                     | Display this help menu                                                      |
| -L|--proxy-port               | Default Proxy Listening Port                                                |
| -s|--offline-service          | Path to the tar.gz service images offline installer                         |
| -x|--offline-plugin           | Path to the tar.gz plugin images offline installer                          |
| -S|--role                     | Role to Upgrade: [app|db|agent|remote_db|guac|proxy]                        |
| -p|--public-hostname          | Agent/Component <IP/Hostname> used to register with deployment.             |
| -g|--database-master-user     | Database master username required for remote DB                             |
| -G|--database-master-password | Database master password required for remote DB                             |
| -q|--db-hostname              | Database Hostname needed when upgrading agent and pulling images            |
| -T|--db-port                  | Database port needed when upgrading agent and pulling images (default 5432) |
| -Q|--db-password              | Database Password needed when upgrading agent and pulling images            |
| -b|--no-check-disk            | Do not check disk space                                                     |
| -n|--api-hostname             | Set API server hostname                                                     |
| -A|--enable-lossless          | Enable lossless streaming option (1.12 and above)                           |
| -O|--use-rolling-images       | Use rolling Service images                                                  |
| -k|--registration-token       | Register a component with an existing deployment.                           |
| --skip-egress                 | Do not install egress network plugin or dependancies                        |
---------------------------------------------------------------------------------------------------------------

Automatic Upgrade Troubleshooting

The upgrade.sh script creates a log file as it runs, this file is removed upon completion of a successful upgrade. However, if something does go wrong the logfile will be available from the directory the upgrade.sh script was executed from in the format kasm_upgrade_${TIMESTAMP}.log. This file will be important for diagnosing the error that caused the upgrade to fail and will be requested when submitting a support ticket with Kasm Technologies.

Update Custom Workspaces

Each release, the Kasm Technologies team updates the default workspaces with new features and security updates. Old workspaces should be removed as they may not be compatible with new Kasm features.

Note

Kasm Workspaces 1.13.0 and newer utilize a Workspaces Registry to install new Workspaces after upgrade. Upon logging into Kasm Workspaces for the first time after the upgrade the administrator will have the opportunity to add the default Kasm Workspaces registry if the administrator has not already configured it.

Workspaces from prior releases may contain incompatible features and are not guaranteed to work. Kasm recommends that the administrator utilize the registry to add the latest compatible Workspaces for the installed version.

Kasm recommends rebuilding any custom Workspaces using the appropriately tagged Docker image (e.g : kasmweb/core-ubuntu-focal:1.16.1 ) see creating custom Kasm images for more information.

See Default and Custom Docker Images for details.

For existing Workspaces officially supported by Kasm, it is possible to edit the Workspace configuration and change the tag to the new version of Kasm Workspaces, this will preserve configuration and group settings that may already have been added. It will take a few minutes for the Kasm Agent to download the new Workspace during which time it will be unavailable.