Understanding Kubernetes Controllers

We will go through different types of controllers: Replication Controllers, ReplicaSet, Deployments. We also see how some operation tasks like autoscaling or rolling updates can be achieved using Kubernetes.

Replication Controllers

They supervise that a number of pods are up and available across all your nodes. On failures pod are replaced by new ones instead of trying to run new ones. You can declare these controllers in YAML files where we specify the image and the number of replicas desired.

_config.yml

Let’s create a Replication controller file (busybox.yml ) to spin up a set of busybox containers.

apiVersion: v1
kind: ReplicationController
metadata:
  name: busybox
spec:
  replicas: 4
  selector:
    app: busybox
  template:
    metadata:
      name: busybox
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        command: [ "/bin/sh", "-c" ]
        args: [ "while true; do sleep 30; done;" ]

In this file we are specifying that the controller should ensure that 4 containers are runnung at every time. Note that I added a command to keep the busybox container alive forever. This is clearly an anti-pattern, but used for simplicity. Every pod will be labeled with the label “busybox” so later we can query K8 which are the pods attached to a certain label.

If you read my post 3. Understanding Kubernetes pods. you can see that the template section is exactly the same that we used when defining a pod descriptor.

» kubectl create -f ./busybox.yml  
replicationcontroller "busybox" created

Once we create a controller, some time is required to provision and download the images for the pods you want to supervise before they start running.

» kubectl describe replicationcontrollers/busybox
Name:		busybox
Namespace:	default
Selector:	app=busybox
Labels:		app=busybox
Annotations:	<none>
Replicas:	4 current / 4 desired
Pods Status:	4 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:	app=busybox
  Containers:
   busybox:
    Image:	busybox
    Port:
    Command:
      /bin/sh
      -c
    Args:
      while true; do sleep 30; done;
    Environment:	<none>
    Mounts:		<none>
  Volumes:		<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  30s		30s		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-54sq9
  30s		30s		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-ptgkc
  30s		30s		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-fpnst
  30s		30s		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-6fxjn

We can check if the controller is doing its job by killing one of the pods under supervision.

» kubectl delete pods busybox-6fxjn
pod "busybox-6fxjn" deleted

Tip: In occassion we may find that killing a rogue pod gives a headache so I recommend using the force flag to get rid of them with a gracefu period

» kubectl delete pods busybox-6fxjn --grace-period=0 --force

Theoretically the replication controller will try to spin up a new one in order to achieve the minimum threshold declared in the replicas section of our file, that was 4.

» kubectl describe replicationcontrollers/busybox | grep replication
  13m		13m		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-54sq9
  13m		13m		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-ptgkc
  13m		13m		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-fpnst
  13m		13m		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-6fxjn
  1m		1m		1	replication-controller			Normal		SuccessfulCreate	Created pod: busybox-v0qpq

The output may be a bit confusing as we see 5 pods. However the controller should not be able to see any longer the pod we killed (The second column indicates that busybox-v0qpq was Last seen 1 minute.

To be 100% sure we can see the pods attached by label. Remember we added the label “busybox” in our controller YAML file?

» kubectl get pods --selector=app=busybox --output=jsonpath={.items..metadata.name}
busybox-54sq9 busybox-fpnst busybox-ptgkc busybox-v0qpq%

So yes the pod we killed (busybox-6fxjn) now is dead and not under the supervision of the controller.

When working with controllers we may be interested on killing all the pods associated to a controller or just kill the controller and leave the pods running on its own under no supervision ( adding the –cascade flag = false). We will clean up everything, the controller and its pods.

» kubectl delete -f ./busybox.yml                                                                           
replicationcontroller "busybox" deleted
» kubectl get pods --selector=app=busybox --output=jsonpath={.items..metadata.name}

The empty output means that there are no pods labeled with “busybox” any longer.

In the case you want to use more advanced patterns like Rolling Updates it needs to be done manually by creating a new Controllers and tweak the replica parameters accordingly ( decreasing in the old one and increasing in the new one). For a detailed understanding see (see Performing rolling updates)

That is the reason why K8 guides recommended a better approach using a Deployment descriptor targeting a ReplicaSet.

Replica Set Controllers

It´s nearly identical to the previous one but with better support for label set selectors. Once a ReplicaSet is in charge of a set of pods we can change the pod label so the ReplicaSet does not takes care of it of any longer.

Autoscaling

Replication Controllers and Replica Set controller can be easily autoscaled based in several conditions. Autoscaling policies can be defined also in yaml files or can be applied on top of a resource to indicate minimum and maximum values of scaling when cpu conditions are reached within our k8 cluster.

_config.yml

We will continue using the previous example. So let´s create again our replicatoin controller. Using the previous example , just starting the controller created 4 pods.

» kubectl create -f ./busybox.yml  
replicationcontroller "busybox" created
» kubectl get pods --selector=app=busybox --output=jsonpath={.items..metadata.name}
busybox-219d6 busybox-bv454 busybox-f5771 busybox-fg8hb

Let´s autoscale the controller between 7 and 10 when the target average CPU utilization (represented as a percent of requested CPU) is over 80%

» kubectl autoscale -f ./busybox.yml  --min=7 --max=10 --cpu-percent=80
» kubectl get pods --selector=app=busybox --output=jsonpath={.items..metadata.name}
busybox-219d6 busybox-bv454 busybox-f5771 busybox-fg8hb busybox-m1rrb busybox-mt698 busybox-zl1hm

In this case as the cpu % usage does not reach the limit the autoscaler does not need to add more container instnaces under its supervision.

Again the recommendation is to use Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don’t require updates at all.

Deployments

Our deployment descriptor looks like

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: busybox
spec:
  replicas: 4
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      name: busybox
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox
        command: [ "/bin/sh", "-c" ]
        args: [ "while true; do sleep 30; done;" ]

I run into multiple issues with my current version of minikube. I tried different apiVersions v1, apps/v1beta1 and apps/v1beta2 with no luck. Always getting the same error

» kubectl create -f ./busybox-deployment.yml                       
Error from server (BadRequest): error when creating "./busybox-deployment.yml": Deployment in version "v1beta2" cannot be handled as a Deployment: no kind "Deployment" is registered for version "apps/v1beta2"

The solution to the issue was Final solution was to restart my minikube cluster with a specific version.

» minikube start --kubernetes-version v1.6.0 --vm-driver=xhyve
» kubectl create -f ./busybox-deployment.yml --record                                
deployment "busybox" created
» kubectl get pods --selector=app=busybox
NAME                      READY   STATUS    RESTARTS   AGE
busybox-361166956-0wq7v  Running  1/1       0          1m
busybox-361166956-28lbj  Running  1/1       0          1m
busybox-361166956-6fn98  Running  1/1       0          1m
busybox-361166956-82z49  Running  1/1       0          1m

We can check the status of the existing Deployments

» kubectl get deployments
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox   4         4         4            4           4m
» kubectl rollout status deployment/busybox
deployment "busybox" successfully rolled out

In the previous controller section we discussed about rolling updates and I mentioned that ReplicaSet supported that approach. It can be accomplished with the following commands to be performed on top of a Deployment descriptor. We are going to downgrade the busybox image to a previous version 1.27.1
Using the edit command we can access to a K8 enhanced version of our deployment file, kept in memory. Locate the image section and update it

» kubectl edit deployment/busybox
deployment "busybox" edited
» kubectl rollout status deployment/busybox
Waiting for rollout to finish: 2 out of 4 new replicas have been updated...
Waiting for rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
deployment "busybox" successfully rolled out

To verify the change we can inspect the Deployment

» kubectl describe deployments          
Name:			busybox
Namespace:		default
CreationTimestamp:	Mon, 04 Dec 2017 07:13:59 +0100
Labels:			app=busybox
Annotations:		deployment.kubernetes.io/revision=2
Selector:		app=busybox
Replicas:		4 desired | 4 updated | 4 total | 4 available | 0 unavailable
StrategyType:		RollingUpdate
MinReadySeconds:	0
RollingUpdateStrategy:	25% max unavailable, 25% max surge
Pod Template:
  Labels:	app=busybox
  Containers:
   busybox:
    Image:	busybox:1.27.1
    .........................
    ..........................

Other useful commands that we may be interested in learning would be being able to undo a rollout change ( in case our deployment went wrong).

» kubectl rollout undo deployment/busybox
deployments "busybox" rolled back

In case of doubt we can look to the history of commands performed to know which is the set of changes happened to a deployment

» kubectl rollout history deployment/busybox
deployments "busybox"
REVISION	CHANGE-CAUSE
1		kubectl create --filename=busybox-deployment.yml --record=true
2		kubectl edit deployment/busybox

Deployments can also scaled, or we can use proportional scaling, but I will let the reader find out how to do it.

Other controller types. Jobs and Daemon Sets

Jobs are used for batch jobs where we want to control whether or not the completion status. Jobs can be comprised of multiple paralell execution and we can specify a backoff limit so the Job knows that all its pods terminated doing its work. Once every pod has finished and the job limit is reached we can consider that the Job itself is complete.

Daemon Sets are used for long lifetime pods related with the nodes itself. Examples can be monitoring a cluster or agregating logs from different containers living in the cluster nodes.

Useful links

The whole blog series about Kubernetes

Written on December 5, 2017