Upgrade cluster's Kubernetes version

Overview

Nebari runs on Kubernetes under the hood, and as administrators of this Kubernetes cluster, one of the maintenance tasks required from time to time is upgrading the version of your Kubernetes cluster. Each of the different cloud providers that Nebari can run on, has their own release cycle and support windows for their flavor of Kubernetes. That said, they tend to follow the official Kubernetes release cycle.

The Nebari development team tries to stay ahead of this by supporting the latest version when possible. However, given that many Kubernetes releases come with a set of deprecations that potentially affect Nebari and downstream plugins, there is an enforced HIGHEST_SUPPORTED_K8S_VERSION allowed.

note

This HIGHEST_SUPPORTED_K8S_VERSION is, at times, a minor version or two behind the officially released Kubernetes version.

Many cloud providers enable users to automatically upgrade their Kubernetes cluster (control plane), however given the potential for deprecations and other changes, Nebari deployed to these clouds has this feature automatically disabled.

This upgrade process bumps the version of the control plane along with all node groups.

warning

Upgrading the kubernetes version of the node groups will cause downtime so please plan accordingly. We also recommend backing up your data before starting this upgrade process.
Kubernetes versions can only be upgraded one minor version at a time. If you're running on 1.24, and need to upgrade to 1.26, you will first need to upgrade to 1.25.
Downgrading to a lower version of Kubernetes is dangerous and we strongly advise against it!

GCP
AWS
Azure
Digital Ocean

Google Kubernetes Engine (GKE) cut their own platform specific version of Kubernetes that usually look something like: 1.26.7-gke.500; this corresponds to a Kubernetes version of 1.26.7.

You can list the supported GKE Kubernetes versions by running the following gcloud command:

gcloud container get-server-config --region <region>

We recommend selecting a validVersion from the STABLE channel:

channels:
- channel: RAPID
  defaultVersion: 1.27.4-gke.900
  validVersions:
  - 1.28.1-gke.201
  ...
- channel: REGULAR
  defaultVersion: 1.27.3-gke.100
  validVersions:
  - 1.27.4-gke.900
  - 1.27.3-gke.1700
  ...
- channel: STABLE
  defaultVersion: 1.27.3-gke.100
  validVersions:
  - 1.27.4-gke.900
  - 1.27.3-gke.100
  - 1.26.7-gke.500
  ...
  - 1.24.15-gke.1700
  - 1.24.14-gke.2700

To upgrade your GKE cluster, update the google_cloud_platform.kubernetes_version field in your nebari-config.yaml to match one of these GKE Kubernetes versions. Then run nebari deploy to apply these changes. This deployment process might take as long as 30 minutes.

info

You will get a validation error if you try to select a Kubernetes version that is unsupported by GKE or a version higher than HIGHEST_SUPPORTED_K8S_VERSION.

Then repeat the above process one minor version at a time. You will get a similar error otherwise:

[terraform]: │ Error: googleapi: Error 400: Master cannot be upgraded to "1.26.7-gke.500": cannot upgrade the master more than a minor version at a time.

For more information about GKE upgrades, please refer to the GKE documentation.

The AWS Elastic Kubernetes Service (EKS) only requires that you supply the major and minor version of Kubernetes that you want. To specify Kubernetes version 1.26.7, update the amazon_web_services.kubernetes_version to 1.26. Then run nebari deploy to apply these changes. This deployment process might take as long as 30 minutes.

info

You will get a validation error if you try to select a Kubernetes version that is unsupported by EKS or a version higher than HIGHEST_SUPPORTED_K8S_VERSION.

In AWS, upgrading EKS will upgrade the control plane components but the node groups will need to be upgraded manually.

In the AWS console, navigate to EKS and click on the name of your Kubernetes cluster (format will be {project-name}-{namespace}). In the 'Compute' tab, scroll down to "Node Groups". Any node groups which are behind will have an "Update Now" button by the "AMI release version" column values. Click "Update Now" for each. Each update may take 15 or more minutes depending on how many workloads need to be migrated, but they can be run simultaneously.

Then repeat the above process one minor version at a time.

For more information about EKS upgrades, please refer to the EKS documentation.

The Azure Kubernetes Service (AKS) requires that you specify the major, minor, and patch version you would like to use. To specify a Kubernetes version update the azure.kubernetes_version to 1.26.7 (or the version you need to upgrade to). Then run nebari deploy to apply these changes. This deployment process might take as long as 30 minutes.

info

You will get a validation error if you try to select a Kubernetes version that is unsupported by AKS or a version higher than HIGHEST_SUPPORTED_K8S_VERSION.

Then repeat the above process one minor version at a time.

For more information about AKS upgrade, please refer to the AKS documentation.

Digital Ocean Kubernetes Service (DOKS) cut their own platform specific version of Kubernetes that usually look something like: 1.25.14-do.0; this corresponds to a Kubernetes version of 1.25.14.

You can list the supported DOKS Kubernetes versions by running the following doctl command:

doctl kubernetes options versions

Slug            Kubernetes Version    Supported Features
28.2-do.0     1.28.2                cluster-autoscaler, docr-integration, ha-control-plane, token-authentication
27.6-do.0     1.27.6                cluster-autoscaler, docr-integration, ha-control-plane, token-authentication
26.9-do.0     1.26.9                cluster-autoscaler, docr-integration, ha-control-plane, token-authentication
25.14-do.0    1.25.14               cluster-autoscaler, docr-integration, ha-control-plane, token-authentication

To upgrade your DOKS cluster, update the digital_ocean.kubernetes_version field in your nebari-config.yaml to match one of these DOKS Kubernetes versions. Then run nebari deploy to apply these changes. This deployment process might take as long as 30 minutes.

info

You will get a validation error if you try to select a Kubernetes version that is unsupported by DOKS or a version higher than HIGHEST_SUPPORTED_K8S_VERSION.

Then repeat the above process one minor version at a time.

For more information about DOKS upgrade, please refer to the DOKS documentation.

Overview​

Overview