[Feature Request] Suppot kubelet and k3s config files when adding a new node #365

New issue

Open

opened 2025-08-19 21:06:56 +00:00 by cloudymax · 0 comments

cloudymax commented

2025-08-19 21:06:56 +00:00

(Migrated from github.com)

Describe the feature you'd like, and why

When adding a new node there are some important kubelet and configuration options we should allow the user to specify:

K3s config file

This file is where we can tell the node:

What IP address to use when communicating with the control-plane eg: normal eth0, VPN device wg0, a bridge at br0 etc...
Specify a path the the kubelet config file
Specify a new default data directory - this prevents k3s from using the system's / disk as the primary storage location which can be problematic in most production servers since it is a very common practice to install the OS on a smaller, slower SATA SSD and make a larger RAID disk available via a mount.

Without changing this option nodes can become tainted due to disk-pressure building on / or develop high levels of CPU throttling waiting for IO on the slow disk.

Example config:

/etc/rancher/k3s/config.yaml

node-ip: 192.168.2.218
bind-address: 192.168.2.218
node-external-ip: 192.168.2.218
kubelet-arg:
- config=/etc/kubernetes/kubelet.yaml
data-dir: /mnt/raid1/rancher/k3s

Kubelet config

The kubelet config lets us enable some very important options:

cpuManagerPolicy and its cpuManagerPolicyOptions allows us to pin cpu cores which improve performance by preventing uneeded rescheduling of tasks to new vcores. It also lets us keep workloads in the same core-group on machines with more than one CPU. Very helpful for streaming apps, VPNs, and latency sensitive workloads.
kubeReserved and systemReserved which allow us to specify an amount of CPU/RAM/Storage that is explicitly reserved for the kubelet and for the host-os. This prevents hardware lock-ups if a app without proper limits tries to eat all the resources on the node.
evictionHard lets us specify an minimum-possible amount of CPU/RAM/Storage which - when matched - will prevent new pod from being scheduled to that node as well as hard-evict offending pods. This also is designed to prevent hardware lock-ups during high loads.
node-labels, node-taints we can also set labels and taints here which is helpful for configuring nodes to run specialized apps that need GPUs, zigbee etc...

Example file:

/etc/kubernetes/kubelet.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerReconcilePeriod: 0s
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"
  distribute-cpus-across-numa: "true"
  align-by-socket: "true"
kubeReserved:
  cpu: "1"
  memory: "1Gi"
  ephemeral-storage: "1Gi"
systemReserved:
  cpu: "1"
  memory: "1Gi"
  ephemeral-storage: "1Gi"
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
featureGates:
  CPUManagerPolicyAlphaOptions: true
node-labels:
  node-role.kubernetes.io/worker: true
node-taints: 
  node-role.kubernetes.io/worker: "NoSchedule"

## Describe the feature you'd like, and why When adding a new node there are some important kubelet and configuration options we should allow the user to specify: ## K3s config file This file is where we can tell the node: 1. What IP address to use when communicating with the control-plane eg: normal eth0, VPN device wg0, a bridge at br0 etc... 2. Specify a path the the kubelet config file 3. Specify a new default data directory - this prevents k3s from using the system's `/` disk as the primary storage location which can be problematic in most production servers since it is a very common practice to install the OS on a smaller, slower SATA SSD and make a larger RAID disk available via a mount. Without changing this option nodes can become tainted due to disk-pressure building on `/` or develop high levels of CPU throttling waiting for IO on the slow disk. Example config: `/etc/rancher/k3s/config.yaml` ```yaml node-ip: 192.168.2.218 bind-address: 192.168.2.218 node-external-ip: 192.168.2.218 kubelet-arg: - config=/etc/kubernetes/kubelet.yaml data-dir: /mnt/raid1/rancher/k3s ``` ## Kubelet config The kubelet config lets us enable some very important options: 1. `cpuManagerPolicy` and its `cpuManagerPolicyOptions` allows us to pin cpu cores which improve performance by preventing uneeded rescheduling of tasks to new vcores. It also lets us keep workloads in the same core-group on machines with more than one CPU. Very helpful for streaming apps, VPNs, and latency sensitive workloads. 2. `kubeReserved` and `systemReserved` which allow us to specify an amount of CPU/RAM/Storage that is explicitly reserved for the kubelet and for the host-os. This prevents hardware lock-ups if a app without proper limits tries to eat all the resources on the node. 3. `evictionHard` lets us specify an minimum-possible amount of CPU/RAM/Storage which - when matched - will prevent new pod from being scheduled to that node as well as hard-evict offending pods. This also is designed to prevent hardware lock-ups during high loads. 4. `node-labels`, `node-taints` we can also set labels and taints here which is helpful for configuring nodes to run specialized apps that need GPUs, zigbee etc... Example file: `/etc/kubernetes/kubelet.yaml` ```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cpuManagerReconcilePeriod: 0s cpuManagerPolicy: static cpuManagerPolicyOptions: full-pcpus-only: "true" distribute-cpus-across-numa: "true" align-by-socket: "true" kubeReserved: cpu: "1" memory: "1Gi" ephemeral-storage: "1Gi" systemReserved: cpu: "1" memory: "1Gi" ephemeral-storage: "1Gi" evictionHard: memory.available: "500Mi" nodefs.available: "10%" featureGates: CPUManagerPolicyAlphaOptions: true node-labels: node-role.kubernetes.io/worker: true node-taints: node-role.kubernetes.io/worker: "NoSchedule" ```

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

small-hack/smol-k8s-lab#365

No description provided.

Rows
Columns