3.2 Time-slice a node¶

Time: 5 min Changes: your Lab 1 node goes from nvidia.com/gpu: 1 to nvidia.com/gpu: 4

Time-slicing is configured in two places: a ConfigMap that describes the sharing policy, and a node label that opts a node into it. The ClusterPolicy connects the two. Nothing changes on a node until it's labelled — which is exactly the control you want in a mixed fleet.

modules/03-sharing/time-slicing-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  # The key name is what you reference from a node label:
  #   kubectl label node <node> nvidia.com/device-plugin.config=four-way
  #
  # Two keys on purpose. The device plugin's config manager has a "single"
  # fallback: a ConfigMap with exactly ONE key is applied to every node,
  # label or not. An explicit no-sharing config keeps unlabelled nodes
  # exclusive and gives you a value to label them with if you want to be
  # explicit.
  exclusive: |-
    version: v1
    flags:
      migStrategy: none
  four-way: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        # Keep the resource name nvidia.com/gpu. With true, the shared
        # resource is advertised as nvidia.com/gpu.shared instead, which is
        # a good production pattern (workloads opt in explicitly).
        renameByDefault: false
        # A pod asking for 2 "GPUs" on a time-sliced node would get 2 slices
        # of the same GPU and think it has two devices. Fail loudly instead.
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

replicas: 4 means the device plugin advertises four nvidia.com/gpu for every physical GPU. failRequestsGreaterThanOne: true rejects any pod asking for 2 on a sliced node — two slices of the same card is not two GPUs, and the pod would find that out the hard way.

Why two keys?

The device plugin's config manager has a single fallback: if the ConfigMap has exactly one key, it applies it to every node whether labelled or not. A ConfigMap with only four-way would slice your whole fleet the moment the Operator picked it up. The exclusive key exists so that unlabelled nodes stay exclusive.

kubectl apply -f modules/03-sharing/time-slicing-config.yaml

Step 2 — Tell the Operator about it¶

kubectl patch clusterpolicies.nvidia.com/cluster-policy -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": ""}}}}'

clusterpolicy.nvidia.com/cluster-policy patched

name points at the ConfigMap; default: "" plus the two-key ConfigMap means no sharing applies unless a node asks for it by label. (Setting default: four-way would slice every GPU node in the cluster — sometimes what you want, never by accident.)

The Operator now re-renders the device-plugin DaemonSet with a config-manager sidecar that watches node labels. You'll see the plugin pod restart once on your GPU node:

kubectl -n gpu-operator get pods -l app=nvidia-device-plugin-daemonset -w

NAME                                   READY   STATUS        RESTARTS   AGE
nvidia-device-plugin-daemonset-7tqrc   1/1     Terminating   0          52m
nvidia-device-plugin-daemonset-b2xk8   0/2     Init:0/1      0          1s
nvidia-device-plugin-daemonset-b2xk8   2/2     Running       0          9s

2/2 — the second container is the config manager. Press Ctrl+C.

The Operator does not watch the ConfigMap

Edit the ConfigMap later (say, replicas: 8) and nothing happens. The device plugin only re-reads its config when the label changes or the pod restarts. NVIDIA's docs say so explicitly. Treat sharing configs as immutable: make a new key (eight-way), then relabel.

Step 3 — Opt your node in¶

export NODE_A=$(kubectl get nodes -l karpenter.sh/nodepool=gpu -o jsonpath='{.items[0].metadata.name}')
echo "$NODE_A"
kubectl label node "$NODE_A" nvidia.com/device-plugin.config=four-way

ip-10-0-23-77.ec2.internal
node/ip-10-0-23-77.ec2.internal labeled

The config manager sees the label, writes the four-way config into place and restarts the plugin container. Ten seconds later:

kubectl get node "$NODE_A" -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu,REPLICAS:.metadata.labels.nvidia\.com/gpu\.replicas,PRODUCT:.metadata.labels.nvidia\.com/gpu\.product,STRATEGY:.metadata.labels.nvidia\.com/gpu\.sharing-strategy'

NAME                         GPU   REPLICAS   PRODUCT            STRATEGY
ip-10-0-23-77.ec2.internal   4     4          NVIDIA-L4-SHARED   time-slicing

Three things changed and all three are visible to the scheduler and to anyone reading labels:

nvidia.com/gpu: 4 — one physical L4, four schedulable units.
nvidia.com/gpu.product: NVIDIA-L4-SHARED — GFD appends -SHARED so a nodeSelector on the unsliced product name will not match this node. That's your safety rail for latency-sensitive workloads: select NVIDIA-L4 and you can never land on a shared card by accident.
nvidia.com/gpu.sharing-strategy: time-slicing — the explicit label if you'd rather select on that.

If the numbers didn't change after 30 seconds, kubectl -n gpu-operator logs -l app=nvidia-device-plugin-daemonset -c config-manager tells you why (usually a typo in the label value vs. the ConfigMap key).

Next: 3.3 Four pods, one GPU →

3.2 Time-slice a node¶

Step 1 — The sharing config¶

Step 2 — Tell the Operator about it¶

Step 3 — Opt your node in¶