Complete Guide to Istio Service Mesh Upgrades

Upgrading Istio in production requires careful planning and execution. This comprehensive guide covers everything you need to know for safe, reliable Istio upgrades.

1. Pre-upgrade Planning and Considerations

Before upgrading Istio, always perform careful planning.

Check Current Istio Version

istioctl version
istioctl version -ojson
kubectl get pods -n istio-system

Review Release Notes

Pay special attention to:

• New features and improvements
• Deprecated or removed fields
• Behavioral changes
• Proxy or sidecar changes
• Breaking changes

Backup Istio Resources and CRDs

kubectl get all -n istio-system -o yaml > istio-backup.yaml
kubectl get crd -o yaml > istio-crd-backup.yaml

Verify Cluster Requirements

• Supported Kubernetes version
• Sufficient cluster resources (CPU, memory)
• Adequate RBAC and permissions
• Networking and CNI plugin compatibility

Plan Downtime and Testing

• Decide if a rolling upgrade is acceptable
• Identify critical workloads
• Schedule maintenance windows if required

2. Prepare istioctl

Install or update istioctl to match the target Istio version:

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=<target-version> sh -
export PATH=$PWD/istio-<target-version>/bin:$PATH

Verify installation:

istioctl version

3. Precheck the Cluster

Before generating manifests or upgrading Istio, run:

istioctl x precheck

Benefits

• Validates Kubernetes version compatibility
• Checks cluster permissions and ServiceAccounts
• Detects missing CRDs or incompatible Istio installations
• Validates CNI and networking readiness

Tip: Always resolve precheck failures before proceeding. This step prevents many common upgrade issues.

4. Generate / Prepare the Manifest

Istio upgrades are typically performed using:

• istioctl install, or
• istioctl manifest generate

Best Practices

Use a custom manifest instead of defaults if you have:

• Private image registries
• Istio CNI enabled
• Custom securityContext for control plane components
• Non-default ingressgateway Service type (ClusterIP, NodePort, or LoadBalancer)

Generic Command Template

istioctl manifest generate \
  --set profile=default \
  --set values.cni.enabled=true \
  --set values.cni.chained=false \
  --set hub=<your-registry> \
  --set tag=<version-tag> \
  --set "values.global.imagePullSecrets[0]=<secret-name>" \
  --set components.pilot.k8s.securityContext.runAsUser=1337 \
  --set components.pilot.k8s.securityContext.runAsGroup=1337 \
  --set components.pilot.k8s.securityContext.runAsNonRoot=true \
  --set values.gateways.istio-ingressgateway.type=<ClusterIP|NodePort|LoadBalancer> \
  > istio-upgrade-manifest.yaml

Example Production Command

./istioctl manifest generate \
  --set profile=default \
  --set values.cni.enabled=true \
  --set values.cni.chained=false \
  --set hub=your-registry.com/istio \
  --set tag=1.20.2 \
  --set "values.global.imagePullSecrets[0]=registry-secret" \
  --set components.pilot.k8s.securityContext.runAsUser=1337 \
  --set components.pilot.k8s.securityContext.runAsGroup=1337 \
  --set components.pilot.k8s.securityContext.runAsNonRoot=true \
  --set values.gateways.istio-ingressgateway.type=ClusterIP \
  > istio-1.20.2-cni-manifest.yaml

Important Notes

• Avoid unsupported fields for proxy containers (newer Istio versions often remove values.global.proxy securityContext fields)
• Always review the generated manifest before applying

5. Dry-run and Review

Always perform a dry-run before applying the manifest:

kubectl apply --dry-run=client -f istio-upgrade-manifest.yaml

Inspect the manifest for:

• Correct securityContext settings
• Desired Service types (ClusterIP / LoadBalancer)
• CNI and sidecar configuration
• Image registry and version tags

6. Apply the Manifest / Upgrade Istio

Apply the manifest:

kubectl apply -f istio-upgrade-manifest.yaml

Recommendation: Use rolling upgrades for the control plane to minimize downtime.

7. Validate Upgrade

Verify Istio Versions

istioctl version -ojson

This confirms:

• Control plane version (istiod)
• Sidecar proxy versions (istio-proxy)
• No version skew between components

Check Istio Pods

kubectl get pods -n istio-system

Verify Istio Services

kubectl get svc -n istio-system

Confirm Installation Health

istioctl verify-install

Functional Validation

• Test application traffic
• Validate ingress and egress paths
• Confirm telemetry, metrics, and tracing

8. Restart Istio Control Plane Pods (If Needed)

Restart control plane components only if required (image, config, or CNI changes):

kubectl rollout restart deployment istiod -n istio-system
kubectl rollout restart deployment istio-ingressgateway -n istio-system
kubectl rollout restart ds istio-cni-node -n kube-system

9. Restart Application / Service Pods (If Needed)

Application pods must be restarted if:

• Sidecar injection template changed
• Proxy image/version changed
• Pod-level securityContext changed

kubectl rollout restart deployment -n <namespace>

Verify new pods have updated sidecars and configuration.

10. Post-upgrade Verification

• Test application traffic through Istio sidecars
• Verify telemetry, tracing, and metrics
• Update CI/CD pipelines with the new Istio version
• Backup the new manifests for rollback purposes

11. Troubleshooting and Rollback

• Use kubectl describe pod and kubectl logs for debugging
• Run istioctl analyze to detect configuration issues
• Rollback by reapplying backed-up manifests or reinstalling the previous Istio version

Best Practices Summary

1. Always back up Istio manifests and CRDs
2. Run precheck before every upgrade
3. Dry-run manifests before applying
4. Use rolling upgrades to minimize downtime
5. Restart pods only when required
6. Validate control plane, proxies, and traffic after upgrade
7. Document changes and maintain upgrade logs

Istio Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                  Kubernetes Cluster                    │
│                                                         │
│  ┌─────────────┐     ┌────────────────────────────┐     │
│  │   istiod    │     │   istio-ingressgateway    │     │
│  │(Control     │     │   (Traffic Entry)         │     │
│  │ Plane)      │     │                           │     │
│  │             │     │ - Envoy Proxy             │     │
│  │ - Pilot     │     │ - ClusterIP/LB/NodePort   │     │
│  │ - Config    │     │                           │     │
│  └──────┬──────┘     └─────────┬─────────────────┘     │
│         │ xDS/Config           │ Traffic               │
│         ▼                      ▼                       │
│  ┌─────────────────────────────────────────────────┐   │
│  │            Application Namespace                │   │
│  │                                                 │   │
│  │  ┌─────────────┐    ┌─────────────┐            │   │
│  │  │  App Pod    │    │  App Pod    │            │   │
│  │  │             │    │             │            │   │
│  │  │ ┌─────────┐ │    │ ┌─────────┐ │            │   │
│  │  │ │istio-   │ │    │ │istio-   │ │            │   │
│  │  │ │proxy    │ │    │ │proxy    │ │            │   │
│  │  │ │(Envoy)  │ │    │ │(Envoy)  │ │            │   │
│  │  │ └─────────┘ │    │ └─────────┘ │            │   │
│  │  └─────────────┘    └─────────────┘            │   │
│  └─────────────────────────────────────────────────┘   │
│                                                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │ Istio CNI (DaemonSet on each node)             │   │
│  │ - Traffic redirection                          │   │
│  │ - No initContainers required                   │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Upgrade Responsibility Matrix

Following this comprehensive guide will ensure your Istio upgrades are smooth, reliable, and maintain high availability for your production workloads.