@kamalmarhubi published his “What even is a kubelet”, the first of three posts on Kubernetes (k8s), in summer of 2015 which introduced basic components of the container orchestration platform one at a time. This is my attempt to update and extend the topic with my own observations and include additional references I have found useful while trying to get my head around Kubernetes.
Why Kubernetes (k8s) and/or Containers
I won’t go much into the “why” of using Kubernetes or containers other than to say if you are asking this question then you might get value out of this post.
What is My Goal Then
My goal is to provide a deeper sense of what each component in a Kubernetes cluster is doing by building up a cluster one component at a time. There is a lot of Kubernetes that can feel like “magic” and it sometimes takes a bit of hands on in isolation to get a feel for what a component is doing.
Arguably the base component is the
kubelet so this post starts there.
If you would rather jump into a running cluster rather than building one up a piece at a time, you will probably find Minikube and the Kubernetes Basics, or even kubeadm, a better fit than this series.
I will be using the current (as of early 2018) Kubernetes 1.9.x default container runtime (Docker) and assume basic familiarity with it and the idea of containers.
For this post, I will assume you are working from a single machine - I was using a Vagrant box on my Mac to run this as I wrote it. (To do so, you may also need Virtual Box if you do not already have it.) I am also assuming you are on a “Linux-like” system. If you are running Windows you might be able to run these commands using the relatively new Windows Subsystem for Linux but I haven’t tried it and Windows has a checkered past when concerning Docker compatibility. See this getting started article for more information on Kubernetes on Windows.
I also chose to use some of the conventions from Kamal’s series - credit to Julia Evans (@b0rk) for the posts leading me to it - and @kelseyhightower’s great Kubernetes The Hard Way. My aspiration is that this is an approachable step-by-step introduction rather than the thorough “speed run” that Kelsey offers. I recommend reading Kelsey’s tutorial when you are ready but it pulls no punches - you have been warned. ;)
Setting Up a Vagrant Box (with Docker and
kubelet, the ultimate focus of this post, assumes you are running
on Linux, or Windows Server 2016 which I don’t have ready access to, I
provisioned an Ubuntu Vagrant
box using the following commands:
From there you will want to
ssh into the Vagrant box and install
Docker using the instructions
on the Ubuntu page
(also copied below). In the instructions the
sudo apt-get update will
probably take a few minutes to complete but ensures you are installing the
latest packages. Also you will probably get asked about using additional
disk at a few of these steps and I answered
Y as I went through this.
That should ultimately result in a “Hello from Docker!” message telling you what it verified on your Docker install. Unless it said there was a problem, you should be good to go for the next steps.
Then we need to grab
kubelet - later it will instruct Docker to run
containers for us.
kubelet (and You)
If Kubernetes were a tree,
kubelet might be its leaves. It is the
code running on nodes in a Kubernetes cluster that actually starts
or stops containers. In many ways it is the only part of Kubernetes
that knows how to do anything about individual containers.
I mentioned that
kubelet will tell Docker to run containers for us.
This is because most of Kubernetes does not see containers on their
own. Instead it cares about “pods” which are collections of 1+
containers that are closely related. An example of this is a web server
in one container and a log manager in another container. You could
manage them independently, but since in many cases you’d want to
manage them as a unit Kubernetes’ use of a “pod” as an abstraction is
Taking a page from
we will use that example here too. You don’t ask Kubernetes to run a
container, you ask it to run a pod - which groups containers - and
kubelet translates that into running containers.
Defining a Pod
Before we can go further, we need to understand how to define a pod.
Like most things in Kubernetes, a YAML file is preferred but JSON is
possible as well. Below is an example of a pod composed of an
container and a log “manager” (really just a truncator) based on a
busybox container - both heavily borrowed from Kamal’s
One thing to note about this config is the shared “volume” between
the two containers. This allows
nginx to write logs and
to manage (i.e. truncate) them in a fairly elegant way.
Save this into a file called
nginx-truncator.yaml in the current directory.
Running and Configuring
There are many ways to configure
kubelet including the following:
- Monitoring a directory of pod manifests
- Polling a URL for pod manifests
- Polling the Kubernetes API
For more information see the
official kublet reference.
I will not go into most of them in this post. Since we are looking at
kubelet in isolation we will use the option for it to watch a
directory for pod manifests to deploy. This is sometimes used for
which are only managed and visible to the
kubelet. This approach is
used by the Kubernetes control plane and described in the
In the next command, we will fire up the
kubelet pointed at a directory
for manifests and let it keep running.
At this time, we need to start a new terminal (leaving the previous one running for now). In that new session, we can check to see what Docker is running, we should see nothing.
Save the pod definition from earlier into a file in the
directory we created above. After a short bit the two containers defined
in the pod should be running.
Side note: unless Hugo and Chroma have figured their formatting out, that code block will look horrible. I’m choosing to ignore it in favor of writing more content despite the desire to fiddle and get the horizontal scrolling CSS working like it’s supposed to. ;)
Update: Some of the formatting was addressed while working on a later post - Syntax Highlighting Problems.
Notice that we have three containers, not just two.
and a container used to store the shared resources across the pod.
As Kamal’s post calls out, we can inspect these containers to see how they are configured and relate to each other.
Only the last container - the one we didn’t specify in the manifest for
the pod - has an IP. This was concerning the first time I saw it because
we are trying to spin up an
nginx instance and what good is that if
we can’t access it.
Reading Kamal’s post further he suggests inspecting the
the containers as well.
So the two containers we specified refer to the third infrastructure
container which references a
NetworkMode. This means that
any ports exposed in the pod are serviced through the IP of the
infrastructure container. Let’s hit port 80 to see if that’s true.
To see that the log truncator is doing it’s job we can use the following to watch the logs:
And in a third, new terminal run the following and see that the second terminal is showing three log events flowing in and then being cleared out.
Talk to the Kube(let)
You can also ask the
kubelet some questions directly via the HTTP
endpoints served up on port 10255.
We have a health check endpoint at
We can get the status of running pods at
We can also see some details of the machine
kubelet is running on
/spec/ (and the trailing
/ is important):
Cleaning Up After Ourselves
When we added a new YAML manifest file to the
kubelet to watch, it created the containers for us. We can
kubelet to remove those same containers by removing that file
curl should indicate no further containers are
A last cleanup step is to
exit from the Vagrant box and
Now that we have a better understanding of what
kubelet brings to
the Kubernetes table, we can move on to layering in things like
the Kubernetes API which I plan to cover in a