Kubernetes By Component - Part 1

@kamalmarhubi published his “What even is a kubelet”, the first of three posts on Kubernetes (k8s), in summer of 2015 which introduced basic components of the container orchestration platform one at a time. This is my attempt to update and extend the topic with my own observations and include additional references I have found useful while trying to get my head around Kubernetes.

Why Kubernetes (k8s) and/or Containers

I won’t go much into the “why” of using Kubernetes or containers other than to say if you are asking this question then you might get value out of this post.

What is My Goal Then

My goal is to provide a deeper sense of what each component in a Kubernetes cluster is doing by building up a cluster one component at a time. There is a lot of Kubernetes that can feel like “magic” and it sometimes takes a bit of hands on in isolation to get a feel for what a component is doing.

Arguably the base component is the kubelet so this post starts there. :)

If you would rather jump into a running cluster rather than building one up a piece at a time, you will probably find Minikube and the Kubernetes Basics, or even kubeadm, a better fit than this series.

I will be using the current (as of early 2018) Kubernetes 1.9.x default container runtime (Docker) and assume basic familiarity with it and the idea of containers.

Getting Started

For this post, I will assume you are working from a single machine - I was using a Vagrant box on my Mac to run this as I wrote it. (To do so, you may also need Virtual Box if you do not already have it.) I am also assuming you are on a “Linux-like” system. If you are running Windows you might be able to run these commands using the relatively new Windows Subsystem for Linux but I haven’t tried it and Windows has a checkered past when concerning Docker compatibility. See this getting started article for more information on Kubernetes on Windows.

I also chose to use some of the conventions from Kamal’s series - credit to Julia Evans (@b0rk) for the posts leading me to it - and @kelseyhightower’s great Kubernetes The Hard Way. My aspiration is that this is an approachable step-by-step introduction rather than the thorough “speed run” that Kelsey offers. I recommend reading Kelsey’s tutorial when you are ready but it pulls no punches - you have been warned. ;)

Setting Up a Vagrant Box (with Docker and kubelet)

Since kubelet, the ultimate focus of this post, assumes you are running on Linux, or Windows Server 2016 which I don’t have ready access to, I provisioned an Ubuntu Vagrant box using the following commands:

1
2

$ vagrant init ubuntu/artful64
$ vagrant up

From there you will want to ssh into the Vagrant box and install Docker using the instructions on the Ubuntu page (also copied below). In the instructions the sudo apt-get update will probably take a few minutes to complete but ensures you are installing the latest packages. Also you will probably get asked about using additional disk at a few of these steps and I answered Y as I went through this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

$ vagrant ssh
$ sudo apt-get update
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    jq \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK
$ sudo apt-key fingerprint 0EBFCD88
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22 [S]

$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce
$ docker --version
Docker version 17.12.0-ce, build c97c6d6
$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete
Digest: sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b75
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

That should ultimately result in a “Hello from Docker!” message telling you what it verified on your Docker install. Unless it said there was a problem, you should be good to go for the next steps.

Then we need to grab kubelet - later it will instruct Docker to run containers for us.

1
2
3
4

$ wget -q --show-progress --https-only --timestamping \
https://storage.googleapis.com/kubernetes-release/release/v1.9.2/bin/linux/amd64/kubelet
kubelet               100%[================================>] 140.95M  16.MB/s   in 9.1s
$ chmod +x kubelet

Containers, Pods, kubelet (and You)

If Kubernetes were a tree, kubelet might be its leaves. It is the code running on nodes in a Kubernetes cluster that actually starts or stops containers. In many ways it is the only part of Kubernetes that knows how to do anything about individual containers.

I mentioned that kubelet will tell Docker to run containers for us. This is because most of Kubernetes does not see containers on their own. Instead it cares about “pods” which are collections of 1+ containers that are closely related. An example of this is a web server in one container and a log manager in another container. You could manage them independently, but since in many cases you’d want to manage them as a unit Kubernetes’ use of a “pod” as an abstraction is helpful.

Taking a page from Kamal’s series we will use that example here too. You don’t ask Kubernetes to run a container, you ask it to run a pod - which groups containers - and kubelet translates that into running containers.

Defining a Pod

Before we can go further, we need to understand how to define a pod. Like most things in Kubernetes, a YAML file is preferred but JSON is possible as well. Below is an example of a pod composed of an nginx container and a log “manager” (really just a truncator) based on a busybox container - both heavily borrowed from Kamal’s original post.

One thing to note about this config is the shared “volume” between the two containers. This allows nginx to write logs and busybox to manage (i.e. truncate) them in a fairly elegant way.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
    volumeMounts:
    - mountPath: /var/log/nginx
      name: nginx-logs
  - name: log-truncator
    image: busybox
    command:
    - /bin/sh
    args: [-c, 'while true; do cat /dev/null > /logdir/access.log; sleep 10; done']
    volumeMounts:
    - mountPath: /logdir
      name: nginx-logs
  volumes:
  - name: nginx-logs
    emptyDir: {}

Save this into a file called nginx-truncator.yaml in the current directory.

1

$ wget https://github.com/joshuasheppard/k8s-by-component/raw/master/part1/nginx-truncator.yaml

Running and Configuring kubelet

There are many ways to configure kubelet including the following:

• Monitoring a directory of pod manifests
• Polling a URL for pod manifests
• Polling the Kubernetes API

For more information see the official kublet reference. I will not go into most of them in this post. Since we are looking at kubelet in isolation we will use the option for it to watch a directory for pod manifests to deploy. This is sometimes used for static pods which are only managed and visible to the kubelet. This approach is used by the Kubernetes control plane and described in the kubeadm init docs.

In the next command, we will fire up the kubelet pointed at a directory for manifests and let it keep running.

1
2

$ mkdir manifests
$ sudo ./kubelet --pod-manifest-path=$PWD/manifests

At this time, we need to start a new terminal (leaving the previous one running for now). In that new session, we can check to see what Docker is running, we should see nothing.

1
2

$ vagrant ssh
$ sudo docker ps

Save the pod definition from earlier into a file in the manfifests/ directory we created above. After a short bit the two containers defined in the pod should be running.

1
2
3
4
5

$ sudo docker ps
CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS              PORTS               NAMES
6fe50599b6d5        busybox                                    "/bin/sh -c 'while t…"   15 minutes ago      Up 15 minutes                           k8s_log-truncator_nginx-ubuntu-artful_default_daafef244e7c2caef984ce760212a72e_0
e8ea4db694b9        nginx                                      "nginx -g 'daemon of…"   15 minutes ago      Up 15 minutes                           k8s_nginx_nginx-ubuntu-artful_default_daafef244e7c2caef984ce760212a72e_0
d310dd99cc96        gcr.io/google_containers/pause-amd64:3.0   "/pause"                 15 minutes ago      Up 15 minutes                           k8s_POD_nginx-ubuntu-artful_default_daafef244e7c2caef984ce760212a72e_0

Side note: unless Hugo and Chroma have figured their formatting out, that code block will look horrible. I’m choosing to ignore it in favor of writing more content despite the desire to fiddle and get the horizontal scrolling CSS working like it’s supposed to. ;)

Update: Some of the formatting was addressed while working on a later post - Syntax Highlighting Problems.

Notice that we have three containers, not just two. nginx, busybox, and a container used to store the shared resources across the pod.

As Kamal’s post calls out, we can inspect these containers to see how they are configured and relate to each other.

1
2
3
4
5
6

$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress  }}' 6fe50599b6d5

$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress  }}' e8ea4db694b9

$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress  }}' d310dd99cc96
172.17.0.2

Only the last container - the one we didn’t specify in the manifest for the pod - has an IP. This was concerning the first time I saw it because we are trying to spin up an nginx instance and what good is that if we can’t access it.

Reading Kamal’s post further he suggests inspecting the NetworkMode of the containers as well.

1
2
3
4
5
6

$ sudo docker inspect --format '{{ .HostConfig.NetworkMode  }}' 6fe50599b6d5
container:d310dd99cc960d5716827e34c45aa56835da3714cf5a7c5f05ed26ec5230bb7e
$ sudo docker inspect --format '{{ .HostConfig.NetworkMode  }}' e8ea4db694b9
container:d310dd99cc960d5716827e34c45aa56835da3714cf5a7c5f05ed26ec5230bb7e
$ sudo docker inspect --format '{{ .HostConfig.NetworkMode  }}' d310dd99cc96
default

So the two containers we specified refer to the third infrastructure container which references a default NetworkMode. This means that any ports exposed in the pod are serviced through the IP of the infrastructure container. Let’s hit port 80 to see if that’s true.

1
2
3
4
5

$ curl --stderr /dev/null http://172.17.0.2 | head -4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

To see that the log truncator is doing it’s job we can use the following to watch the logs:

1

$ sudo docker exec -t 6fe50599b6d5 cat /logdir/access.log

And in a third, new terminal run the following and see that the second terminal is showing three log events flowing in and then being cleared out.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

$ vagrant ssh
$ curl --stderr /dev/null http://172.17.0.2 | head -4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ curl --stderr /dev/null http://172.17.0.2 | head -4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ curl --stderr /dev/null http://172.17.0.2 | head -4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Talk to the Kube(let)

You can also ask the kubelet some questions directly via the HTTP endpoints served up on port 10255.

We have a health check endpoint at /healthz:

1

$ curl http://localhost:10255/healthz

We can get the status of running pods at /pods:

1

$ curl --stderr /dev/null http://localhost:10255/pods | jq . | head

We can also see some details of the machine kubelet is running on using /spec/ (and the trailing / is important):

1

$ curl --stderr /dev/null  http://localhost:10255/spec/ | jq . | head

Cleaning Up After Ourselves

When we added a new YAML manifest file to the manifests/ directory we asked kubelet to watch, it created the containers for us. We can get kubelet to remove those same containers by removing that file as well.

1
2
3
4

$ rm $PWD/manifests/nginx-truncator.yaml
$ sleep 20  # wait for the kublet to spot the removed manifest
$ sudo docker ps
$ curl --stderr /dev/null http://localhost:10255/pods

Both the ps and curl should indicate no further containers are running.

A last cleanup step is to exit from the Vagrant box and destroy it.

1
2

$ exit
$ vagrant destroy

What’s Next

Now that we have a better understanding of what kubelet brings to the Kubernetes table, we can move on to layering in things like the Kubernetes API which I plan to cover in a future post.