Making Karpenter behave

To understand how to make Karpenter behave, it's useful to understand why Karpenter exists in the first place.

Kubernetes has always been designed around orchestrating containers across multiple nodes (hosts/virtual machines, etc.), including handling the concept of nodes going offline (maintenance or a fault) and new nodes coming online. With Kubernetes finally arriving as a first-class citizen at the Hyperscalers (AWS, Google Cloud, Azure, etc.), the concept of autoscaling nodes quickly took off.

However, in complex topologies such as multiple availability zones (AZs) in AWS, this caused undesired issues like node awareness based on where an EBS (storage volume) exists and pod scheduling to match that. Cluster Autoscaler has done a great job of scaling clusters based on CPU and memory utilisation, but very naively. It would simply add a new node, mostly trying to balance its distribution across availability zones (not always).

What this inadvertently causes is scheduling capacity, but with unmet constraints. In AWS, an EBS volume is tied to an AZ, and thus capacity in other AZs won't help a pod that requires that volume - it would get scheduled but would fail.

In other scenarios, you'd have topology spread constraints (marked as required) which cannot be scheduled because there is not enough capacity in the topology that would meet the spread requirements. Now, before someone mentions it, you can, of course, not make the topology spread constraint required, but then you're basically living in a lie that your workload meets your spread objectives - so what's the point?

To solve this, Karpenter was born. Consider it a more advanced cluster autoscaler.

Karpenter looks for pending pods, understands their scheduling constraints (volume placement, topology spread, etc.) and then orchestrates the provisioning of a node (typically one that is the right size) in the correct zone. What I like to call this is just-in-time (JIT) node provisioning. To be clear, you want that pending pod status, which is what Karpenter reacts to in order to make decisions.

At this point, you may point out that it sounds like Karpenter is behaving just fine. There are, however, additional features of Karpenter which typically cause workload instability if they're not designed well. There are three main features of Karpenter that directly and indirectly cause nodes to cycle.

1. Consolidation - when Karpenter makes the initial scheduling decision, it's to handle the scheduling of a pending pod. Over time, this could lead to multiple smaller nodes, which may be better placed on one larger node (EBS and topology spread constraints considered). Karpenter then provisions the new node and drains the older, smaller nodes (by default, Kubernetes cordons them as well), and the pods (almost instantly depending on their configuration) start on the new single node.
2. Node Expiry - to ensure nodes stay healthy and that they get the latest patched EKS AMIs (Amazon Machine Images), Karpenter expires nodes after a configurable time period and introduces a new node to move the workload to. Typically it's the same or similar node, but Karpenter always tries to provision the cheapest node.
3. Spot Instances - this is one of my favourites. Karpenter can provision either spot or on-demand instances. I am writing a post regarding spot instances, especially in production. If you don't know what they are, subscribe for updates on that post (yes, I did just ask you to subscribe). If a spot instance needs to terminate, then Karpenter handles that event and provisions a new node.

In all of these cases, Karpenter is provisioning a new node and then attempting to drain nodes so pods can quickly move to the new nodes. If teams haven't configured their workloads correctly (both at an app level and a Kubernetes level), then they'd likely see a disruption.

To handle disruption and ensure workloads handle it correctly with zero outage, you have to maintain certain standards. So when we want Karpenter to behave, we're actually wanting to have workloads behave correctly in response to these disruption events.

Ensure Topology Spread Constraints are set

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    minDomains: 3 # 3 AZs
    labelSelector:
      matchLabels:
        app: my-workload

Configure Topology Spread Constraints and ensure that your pods at least spread across different nodes, but preferably across zones. Typically, you would not attempt to spread away from a host and an availability zone at the same time, as you will just get many small nodes. This makes sure that replicas of your workload exist away from a single node getting drained, and thus you have other pods ready to receive traffic.

Make sure that your apps handle graceful termination

Most frameworks and application servers already handle this, but that is not always the case. When Kubernetes drains a node (invoked via Karpenter), it sends a graceful termination signal to each pod on the node. If it's handled correctly, the app first handles open connections and processes before actually deleting the pod. At the same time, a replacement pod is already provisioned elsewhere and starts accepting new traffic as soon as it's ready.

Lighter containers allow for more agile workloads

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o app .

FROM gcr.io/distroless/static-debian12
COPY --from=build /src/app /app
ENTRYPOINT ["/app"]

Creating a container as small as possible reduces the time it takes to pull onto the node, so a container being ready to start is significantly faster. When images are larger, the time for a new pod to be ready increases, which means all traffic is being handled by n-2 (two fewer) pods, which could cause degradation on that workload.

Make sure your health probes are configured

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Configure probes on your workloads. Kubernetes marks pods as ready to receive traffic based on health probes (specifically readiness probes). When these are not configured, pods are marked as ready and automatically receive new traffic even if the app itself isn't ready to receive the traffic. This is often a problem I see in Kubernetes clusters (of all kinds).

Set up a Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-workload-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-workload

Pod disruption budgets signal how many pods you're willing to run without. Karpenter honours this, and it's one of the more important items to handle sweeping node expiry disruptions. Multiple nodes can expire at the same time, causing multiple pods (potentially all of the pods for a workload) to be terminated along with those nodes.

Always have more than 1 replica of your pod

spec:
  replicas: 3
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"  # only for the rare single-pod case

Always have multiple replicas of your pod, and ideally you should run as many as you have zones. To be honest, the more pods you have, the more resilient your workload will be, especially on busy workloads. In the rare case that an application can only run a single pod, you would have to mark the pod as non-disruptable, but this is not ideal and doesn't make the pod immune to node expiry. I would push back hard on the devs to find a way to handle this going forward (either by a queuing system which allows an upstream pod to restart and work to continue or by refactoring the application).

At LSD Open, we run all our production nodes on spot instances and have a number of customers that run full fleets of spot across production estates as well. Only when the items above are missing or misconfigured do we see issues regarding Karpenter and shifting nodes. However, Kubernetes is designed for this, and the primitives mentioned above are how organisations with hundreds of nodes don't really concern themselves with constant node changes below their applications.