--- myst: html_meta: "description lang=en": "Manual and automated upgrade process documentation for Kasm Workspaces multi server deployments." "keywords": "Kasm, Workspace, Multi-server, Multi, Server, Upgrade" "property=og:locale": "en_US" --- ```{title} Multi Server Upgrade ``` # Multi Server Upgrade ```{warning} When running any upgrade please ensure there are no active Kasm Workspaces sessions running. On large deployments this will likely require a maintenance window. ``` ```{note} While not required, making a backup using a VM snapshot or other method of the servers in the installation is recommended as a precaution to provide a recovery point if something goes wrong. ``` ## Manually Please read through the entire process before getting started. Issues can be reported on the [Kasm Workspaces Issues Page](https://github.com/kasmtech/workspaces-issues/issues) ```{include} image_warning.md ``` - Stop all Kasm Services on all Servers ```Bash sudo /opt/kasm/bin/stop ``` ### Upgrade Database Server Role Create a Database Backup - Ensure Kasm services are stopped on all hosts. Database corruption can occur if Kasm services are still connected to the database when it goes down. ```{note} If the database is a standalone remote database refer here for backup instructions. [](../how_to/remote_database.md#backing-up-the-postgresql-server). ``` ```Bash sudo /opt/kasm/bin/stop ``` - Execute the database backup utility ```{parsed-literal} sudo mkdir -p /opt/kasm/backups/ sudo bash /opt/kasm/{{ previous_release }}/bin/utils/db_backup -f /opt/kasm/backups/kasm_db_backup.tar -p /opt/kasm/{{ previous_release }}/ ``` - Verify the presence and location of the database backup ```Bash sudo ls -al /opt/kasm/backups/kasm_db_backup.tar ``` - Download and extract the new installation media ```{parsed-literal} cd /tmp/ curl -O {{ release_url }} tar -xf kasm_release_*.tar.gz ``` ```{note} When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see {doc}`System Requirements <../install/system_requirements>` for details. ``` - Get the existing database password for use in the subsequent commands. ```{parsed-literal} sudo grep " password" /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml ``` - Get the existing redis password for use in the subsequent commands. ```{parsed-literal} sudo grep "redis_password" /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml ``` **Perform a clean install** Perform a clean install of the Database Role Services ```{note} If the database is a standalone remote database refer here for clean install instructions. [](../how_to/remote_database.md#initializing-postgresql-server). ``` - Install kasm from the release media downloaded in the prior steps. - When performing an offline update add these flags `--offline-workspaces --offline-service ` ```Bash sudo bash kasm_release/install.sh -S db -D -Q -R ``` ### Restore and upgrade the database Restore and update the database from the prior version - Ensure all Kasm services are stopped on all hosts ```{note} If the database is a standalone remote database refer here for restoration instructions. [](../how_to/remote_database.md#restoring-the-postgresql-server-from-a-backup). ``` ```Bash sudo /opt/kasm/bin/stop ``` - Execute the database restore command ```{parsed-literal} sudo /opt/kasm/{{ release }}/bin/utils/db_restore -f /opt/kasm/backups/kasm_db_backup.tar -p /opt/kasm/{{ release }}/ ``` - Perform an upgrade of the database schema ```{note} If the database is a standalone remote database refer here for upgrade instructions. [](../how_to/remote_database.md). ``` ```{parsed-literal} sudo /opt/kasm/{{ release }}/bin/utils/db_upgrade -p /opt/kasm/{{ release }} ``` - **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration. ```{parsed-literal} diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml /opt/kasm/{{ release }}/docker/docker-compose.yaml ``` - (Optional) Install the Kasm {{ release }} default images for the platform - With Kasm Workspaces 1.13.0 and newer the Kasm team recommends using the Workspaces Registry to install Workspaces rather than seeding the images during the installation. - Seeding workspaces is still needed when doing an offline Kasm installation as the official Kasm Workspaces Registry will be unavailable. - If doing an offline upgrade first extract the default image seed data from the workspace images tar. For x86 (amd64) platforms: ```{parsed-literal} tar xf --strip-components=1 -C /opt/kasm/{{ release }}/conf/database/seed_data/ workspace_images/default_images_amd64.yaml ``` For ARM (arm64) platforms: ```{parsed-literal} tar xf --strip-components=1 -C /opt/kasm/{{ release }}/conf/database/seed_data/ workspace_images/default_images_arm64.yaml ``` For x86 (amd64) platforms: ```{parsed-literal} sudo /opt/kasm/{{ release }}/bin/utils/db_init -s /opt/kasm/{{ release }}/conf/database/seed_data/default_images_amd64.yaml ``` For ARM (arm64) platforms: ```{parsed-literal} sudo /opt/kasm/{{ release }}/bin/utils/db_init -s /opt/kasm/{{ release }}/conf/database/seed_data/default_images_arm64.yaml ``` - Start the Kasm services ```Bash sudo /opt/kasm/bin/start ``` ### Upgrade Web App Server Role - Stop existing Kasm Services ```Bash sudo /opt/kasm/bin/stop ``` - Download and extract the new installation media ```Bash cd /tmp/ curl -O |release_url| tar -xf kasm_release_*.tar.gz ``` ```{note} When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see {doc}`System Requirements <../install/system_requirements>` for details. ``` - Install kasm from the release media downloaded in the prior steps. - When performing an offline update add this flag `--offline-service ` ```Bash sudo bash kasm_release/install.sh -S app -D -q -Q -R [REDIS_PASSWORD] ``` - Copy **manager_id** and **server_hostname** from the old configuration into the new ```{parsed-literal} grep manager_id /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml grep server_hostname /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml sudo vi /opt/kasm/{{ release }}/conf/app/api.app.config.yaml ``` - **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration. ```{parsed-literal} diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml /opt/kasm/{{ release }}/docker/docker-compose.yaml ``` - Start the Kasm services ```Bash sudo /opt/kasm/bin/start ``` ### Upgrade Agent Server Role - Ensure all Kasm services are stopped ```Bash sudo /opt/kasm/bin/stop ``` - Download and extract the new installation media ```{parsed-literal} cd /tmp/ curl -O {{ release_url }} tar -xf kasm_release_*.tar.gz ``` - Get the existing manager token for use in the subsequent commands. ```{parsed-literal} sudo grep "token" /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml ``` ```{note} When performing an offline upgrade please ensure that versions of docker and docker compose that meet the updated system requirements have been downloaded and installed see {doc}`System Requirements <../install/system_requirements>` for details. ``` - Install kasm from the release media downloaded in the prior steps. - When performing an offline update add these flags `--offline-workspaces --offline-service ` ```Bash sudo bash kasm_release/install.sh -S agent -D -p -m -M ``` - Copy the server_id and the public_hostname properties from the old agent to the new ```{parsed-literal} grep server_id /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml grep public_hostname /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml sudo vi /opt/kasm/{{ release }}/conf/app/agent.app.config.yaml ``` - Copy the auto-generated nginx configs for any sessions that may exist on the Agent ```{parsed-literal} sudo cp /opt/kasm/{{ previous_release }}/conf/nginx/containers.d/* /opt/kasm/{{ release }}/conf/nginx/containers.d/ ``` - **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration. ```{parsed-literal} diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml /opt/kasm/{{ release }}/docker/docker-compose.yaml diff /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml /opt/kasm/{{ release }}/conf/app/agent.app.config.yaml ``` - Start the Kasm services ```Bash sudo /opt/kasm/bin/start ``` - Verify the Agent Service is properly checking into the Manager Service. Log into the UI as an Administrator. Select **Agents** and verify the Agent is listed with a **Last Reported** time of less than 1 minute. If the agent has not checked in, it likely can’t resolve or connect to \[AGENT_HOSTNAME\]:443 . Inspect the logs for details. ```bash sudo tail -f /opt/kasm/current/log/agent.log ``` ### (Optional) Install the Connection Proxy (Guac) Role The Connection Proxy role is new in Workspaces 1.12.0. This service is used to connect to VM/Hardware running RDP, VNC, or SSH. If these capabilities are not needed, this role does not need to be installed. Login to the Workspaces UI as an administrator. Retrieve the value of **Component Registration Token** from the Global Settings. - Download and extract the new installation media ```{parsed-literal} cd /tmp/ curl -O {{ release_url }} tar -xf kasm_release_*.tar.gz ``` - Install the Connection Proxy role from the release media downloaded in the prior steps. - When performing an offline update add these flags `--offline-workspaces --offline-service ` ```Bash sudo bash kasm_release/install.sh --role guac --api-hostname [MANAGER_HOSTNAME] --public-hostname [CONNECTION_PROXY_HOSTNAME] --registration-token [SERVICE_REGISTRATION_TOKEN] -D ``` - Copy the id and the token properties from the old connection_proxy to the new - The *token* field of *kasmguac.app.config.yaml* has been renamed to *auth_token* in the Kasm Workspaces 1.13.0 release ```{parsed-literal} grep id /opt/kasm/{{ previous_release }}/conf/app/kasmguac.app.config.yaml grep token /opt/kasm/{{ previous_release }}/conf/app/kasmguac.app.config.yaml sudo vi /opt/kasm/{{ release }}/conf/app/kasmguac.app.config.yaml ``` ```Bash sudo /opt/kasm/bin/start ``` ## Upgrade Script Since Kasm Workspaces version 1.11.0 an upgrade script is included with the installation package under kasm_release/upgrade.sh. ````{note} Before starting the upgrade process please ensure that the Web App servers are stopped. If left running there is a chance for database corruption on the Webb App servers ```Bash sudo /opt/kasm/bin/stop ``` ```` The upgrade script can be found in the installation package under kasm_release/upgrade.sh. These instructions assume there are 3 servers with the app, db, and agent roles. Login to the db server and execute: `````{tabs} ````{tab} Standard Upgrade ```{parsed-literal} cd /tmp curl -O {{ release_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role db ``` ```` ````{tab} Offline Upgrade x86-64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ amd_service_images_url }} curl -O {{ amd_workspace_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role db --offline-service /tmp/{{ amd_service_images_url.split('/') | last }} ``` ```` ````{tab} Offline Upgrade arm64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ arm_service_images_url }} curl -O {{ arm_workspace_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role db --offline-service /tmp/{{ arm_service_images_url.split('/') | last }} ``` ```` ````` Login to the agent server and execute: `````{tabs} ````{tab} Standard Upgrade ```{parsed-literal} cd /tmp curl -O {{ release_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role agent ``` ```` ````{tab} Offline Upgrade x86-64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ amd_service_images_url }} curl -O {{ amd_workspace_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role agent --offline-service /tmp/{{ amd_service_images_url.split('/') | last }} ``` ```` ````{tab} Offline Upgrade arm64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ arm_service_images_url }} curl -O {{ arm_workspace_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role agent --offline-service /tmp/{{ arm_service_images_url.split('/') | last }} ``` ```` ````` Login to the Web App server and execute: `````{tabs} ````{tab} Standard Upgrade ```{parsed-literal} cd /tmp curl -O {{ release_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role app ``` ```` ````{tab} Offline Upgrade x86-64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ amd_service_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role app --offline-service /tmp/{{ amd_service_images_url.split('/') | last }} ``` ```` ````{tab} Offline Upgrade arm64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ arm_service_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role app --offline-service /tmp/{{ arm_service_images_url.split('/') | last }} ``` ```` ````` Login to the connection proxy server and execute: `````{tabs} ````{tab} Standard Upgrade ```{parsed-literal} cd /tmp curl -O {{ release_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role guac ``` ```` ````{tab} Offline Upgrade x86-64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ amd_service_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role guac --offline-service /tmp/{{ amd_service_images_url.split('/') | last }} ``` ```` ````{tab} Offline Upgrade arm64 ```{parsed-literal} cd /tmp curl -O {{ release_url }} curl -O {{ arm_service_images_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/upgrade.sh --role guac --offline-service /tmp/{{ arm_service_images_url.split('/') | last }} ``` ```` ````` Here are the available helper flags for the upgrade.sh script: ```text Flag Description --------------------------------------------------------------------------------------------------------------- | -h|--help | Display this help menu | | -L|--proxy-port | Default Proxy Listening Port | | -s|--offline-service | Path to the tar.gz service images offline installer | | -S|--role | Role to Upgrade: [app|db|agent|remote_db|guac|proxy] | | -p|--public-hostname | Agent/Component used to register with deployment. | | -g|--database-master-user | Database master username required for remote DB | | -G|--database-master-password | Database master password required for remote DB | | -q|--db-hostname | Database Hostname needed when upgrading agent and pulling images | | -T|--db-port | Database port needed when upgrading agent and pulling images (default 5432) | | -Q|--db-password | Database Password needed when upgrading agent and pulling images | | -b|--no-check-disk | Do not check disk space | | -n|--api-hostname | Set API server hostname | | -A|--enable-lossless | Enable lossless streaming option (1.12 and above) | | -O|--use-rolling-images | Use rolling Service images | | -k|--registration-token | Register a component with an existing deployment. | | --slim-images | Use slim alpine based service containers | --------------------------------------------------------------------------------------------------------------- ``` ### Automatic Upgrade Troubleshooting The upgrade.sh script creates a log file as it runs, this file is removed upon completion of a successful upgrade. However, if something does go wrong the logfile will be available from the directory the upgrade.sh script was executed from in the format *kasm_upgrade_${TIMESTAMP}.log*. This file will be important for diagnosing the error that caused the upgrade to fail and will be requested when submitting a support ticket with Kasm Technologies. ```{include} upgrade_images.md ```