Operations

HA deployments, upgrades, monitoring, troubleshooting.

High Availability

3-server minimum

HA requires an odd number of server nodes (3, 5, 7) for etcd quorum.

Embedded etcd

Each server node runs its own etcd — no external dependency. Use --cluster-init on the first server.

Worker scaling

Add agents with --server and --token. Pool of 10s or 100s of agents is fine.

Load balancing

Put a TCP load balancer in front of the 3 server IPs for kube-apiserver. L4, round-robin is fine.

Upgrades

Manual

curl -sfL https://get.k3s.io | sh - on each server (then agent). One at a time for server nodes.

Automated

Use the system-upgrade controller — see our operations guide for the manifests.

Skipping versions

Generally safe. Check the release notes for breaking changes.

Roll back

Not officially supported. K3s upgrades are deliberately one-way. Take etcd snapshots first.

Monitoring

Built-in metrics

kubelet, kube-state-metrics, node-exporter — all expose Prometheus metrics on :8080/metrics (kubelet).

Integrations

Rancher Monitoring (Helm chart) ships with Grafana dashboards. Or wire up your own Prometheus.

Logs

journalctl -u k3s -f, or k3s kubectl logs -f <pod>.

Backup Strategies

Frequency

Daily for production workloads. Hourly for transactional data (DBs).

Tooling

Velero for full cluster backup. etcdctl snapshot for control plane state.

Test restores

Monthly drill — restore to a test cluster and verify workloads come up.

Troubleshooting

Node NotReady

Check kubelet: journalctl -u k3s | grep kubelet. Verify swap is off. Check disk pressure.

Pod CrashLoopBackOff

kubectl describe pod + kubectl logs. Image pull issues? Check imagePullSecrets.

Network connectivity

Use netshoot. Test DNS: nslookup kubernetes.default. Test connectivity: nc -zv <pod-ip> 80

Where to ask

K3s community Slack, GitHub issues, or local — see our community page.

Air-Gapped Install

1. Prepare the bundle (online machine)

On a machine with internet access, gather the install script, binary, and images:

# Download the install script
curl -sfL https://get.k3s.io -o install.sh

# Get the binary (architecture-specific) — amd64 / arm64 / armv7
curl -fL -o k3s https://github.com/k3s-io/k3s/releases/latest/download/k3s
chmod +x k3s

# Get the image tarball (amd64)
curl -fL -o k3s-airgap-images-amd64.tar \
  https://github.com/k3s-io/k3s/releases/latest/download/k3s-airgap-images-amd64.tar

# Pack them up
tar cf k3s-airgap-bundle.tar install.sh k3s k3s-airgap-images-amd64.tar
# Transfer to the air-gapped target

2. Configure private registry (optional)

If you have a private registry mirror, create the registries config before starting the service:

# /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "https://your.private.registry:5000"
configs:
  "your.private.registry:5000":
    auth:
      username: your_user
      password: your_pass
    tls:
      insecure_skip_verify: true

3. Install on the target

# Unpack the bundle
tar xf k3s-airgap-bundle.tar

# Install binary
sudo cp k3s /usr/local/bin/k3s
sudo chmod +x /usr/local/bin/k3s

# Place image tarball (K3s loads it on startup)
sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/

# Run the install (disables online check)
sudo INSTALL_K3S_SKIP_ENABLE=true sh ./install.sh
sudo systemctl enable --now k3s

# Verify
sudo k3s kubectl get node

4. Optional: System Upgrade Controller

For automated rollouts across air-gap clusters, deploy the system-upgrade controller. Since air-gapped, you'll need to pre-pull its image as well:

# Save controller manifests locally
curl -fL -o suc.yaml https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml
# 注：若离线，可从一台联网机先行下载后镜像到私仓
# Edit the image: rancher/system-upgrade-controller:vX.Y.Z tag → push to your private registry
# Apply the modified manifest
sudo k3s kubectl apply -f suc.yaml

# Define an upgrade plan
sudo k3s kubectl apply -f - <<EOF
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/control-plane: "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  version: v1.30.0+k3s1
EOF

← Back to Docs Index Install Guide →