Upgrade Troubleshooting Runbook

This runbook provides troubleshooting and post-upgrade configuration steps for upgrading Turbine from 26.1.4 to 26.2.3.

It covers:

Resolving the Velero Local Volume Provider (LVP) compatibility issue during the upgrade.

Reconfiguring snapshot storage after the upgrade.

Upgrading KOTS to 1.131.6 to restore the NFS, InternalStorage, and HostPath snapshot configuration options in the Admin Console.

2. Issue: Velero Local Volume Provider Incompatibility

2.1 Upgrade Failure

During the upgrade from 26.1.4 to 26.2.3, the upgrade script may fail with:

✔ In cluster Preflights success

Checking if git is installed

Checking if libzstd is installed

Waiting for minio deployment to rollout

Getting Minio storage format

Velero 1.18.2 cannot be used with the Local Volume Provider
(provider: replicated.com/nfs).

Velero 1.17+ uses Kopia, which does not support the Local Volume Provider.
Migrate the default BackupStorageLocation to an object store
(S3-compatible, Rook, or Minio) before upgrading.

2.2 Root Cause

Turbine 26.2.3 uses Velero 1.18.2.

Velero 1.17 and later use Kopia, which does not support the Local Volume Provider (LVP).

If the existing environment uses LVP-based snapshot storage such as:

InternalStorage

HostPath

the upgrade will fail until the existing LVP-backed Velero configuration is removed.

3. Workaround: Remove Existing LVP Snapshot Configuration

Before proceeding, confirm that the customer understands that the existing LVP-based snapshots will not be available for restore after moving to the new snapshot configuration.

The recommended target configuration is an object-store-compatible backend such as:

S3-compatible storage

Rook

Minio

Run the following commands from a node with kubectl access to the cluster.

3.1 Delete the Existing BackupStorageLocation

kubectl delete backupstoragelocation -n velero default

3.2 Delete Existing BackupRepositories

kubectl delete backuprepository -n velero --all

3.3 Delete the Internal Velero Snapshots PVC

kubectl delete pvc -n velero velero-internal-snapshots

If the PVC does not exist, the following error can be ignored:

Error from server (NotFound):
persistentvolumeclaims "velero-internal-snapshots" not found

3.4 Delete the Velero Deployment

kubectl delete deployment -n velero velero

3.5 Delete the Velero Node Agent

kubectl delete daemonset -n velero node-agent

4. Verify the Old Configuration Is Removed

Run:

kubectl get backupstoragelocation -n velero -oyaml
kubectl get backuprepository -n velero
kubectl get pvc -n velero
kubectl get deployment -n velero velero
kubectl get daemonset -n velero node-agent

Confirm that the old LVP-backed resources are no longer present.

5. Rerun the Upgrade

After removing the old LVP configuration, rerun the 26.1.4 → 26.2.3 upgrade installer.

The installer should now proceed past the Velero compatibility check and continue with the upgrade.

Wait for the installer to complete successfully.

6. Post-Upgrade Snapshot Configuration

6.1 KOTS Version Change

After upgrading to Turbine 26.2.3, the KOTS version is upgraded to 1.130.7.

KOTS 1.130.7 does not provide the Admin Console options required to configure the following snapshot storage types:

InternalStorage

HostPath

Therefore, these options may not be available under the snapshot settings after the Turbine upgrade.

6.2 Required KOTS Version

To make the NFS, InternalStorage, and HostPath snapshot configuration options available again in the Admin Console, manually upgrade KOTS to:

KOTS 1.131.6

7. Manually Upgrade KOTS to 1.131.6

Step 1 – Download the Installer Script

Download the Turbine 26.2.3 installer script manually:

curl -sSL https://kurl.sh/turbine-turbine-26-2-3 > install.sh

Verify that the script was downloaded:

ls -lh install.sh

Step 2 – Edit the Installer Script

Open the installer script using an editor:

vi install.sh

Locate the kotsadm configuration.

The configuration will contain the KOTS version used by the installer.

Update it to 1.131.6:

kotsadm:
    applicationSlug: turbine/turbine-26-2-3
    version: 1.131.6
    s3Override: https://s3.kurl.sh/external/kotsadm-1.131.6.tar.gz

Important

Ensure both values are updated consistently:

version: 1.131.6

and:

s3Override: https://s3.kurl.sh/external/kotsadm-1.131.6.tar.gz

Save the file and exit the editor.

8. Run the Modified Installer

After updating the KOTS version, run the installer again:

cat install.sh | sudo bash -s ha

Allow the installer to complete.

The installer should upgrade KOTS from 1.130.7 to 1.131.6.

9. Verify the KOTS Version

After the installer completes, verify that KOTS has been upgraded successfully.

For example:

kubectl get pods -n kotsadm

You can also verify the KOTS version from the Admin Console.

Confirm that the installed KOTS version is:

1.131.6

10. Configure Snapshot Storage

After confirming that KOTS 1.131.6 is installed:

Navigate to the Snapshot Settings.

Verify that the required snapshot storage options are available.

Select the storage type based on the customer's requirements:

InternalStorage

HostPath

Configure the required storage parameters.

Save the configuration.

11. Complete Upgrade and Configuration Flow

The complete process is:

Turbine 26.1.4
       │
       ▼
Start upgrade to 26.2.3
       │
       ▼
Velero LVP compatibility error
       │
       ▼
Remove old LVP configuration
       │
       ├── BackupStorageLocation
       ├── BackupRepositories
       ├── Internal Velero PVC
       ├── Velero Deployment
       └── Velero Node Agent
       │
       ▼
Rerun 26.2.3 upgrade
       │
       ▼
Upgrade completes
       │
       ▼
KOTS 1.130.7
       │
       ▼
NFS / InternalStorage / HostPath options unavailable
       │
       ▼
Download install.sh manually
       │
       ▼
Update KOTS to 1.131.6
       │
       ▼
Run modified installer
       │
       ▼
KOTS 1.131.6
       │
       ▼
Log in to Admin Console
       │
       ▼
Configure Snapshot Settings
       │
       ▼
Select required storage type