This is part 2 in a series of posts related to
Part 1 was about the
Like part 1, this post is inspired by
another 2015 post
by Kamal Marhubi and is intended to
update and expand on the topics with my own thoughts and learnings. We
kube-apiserver, its relationship to
etcd, and get a
taste of why
kubectl is so nice.
As in the previous post, my goal is to provide a deeper sense of what each component in a Kubernetes cluster is doing by building up a cluster one component at a time. There is a lot of Kubernetes that can feel like “magic” and it sometimes takes a bit of hands on in isolation to get a feel for what a component is doing.
kubelet is the base component, then
kube-apiserver (the Kubernetes
API service) is the next level up. It also serves as the primary
interaction point between each Kubernetes component and the central store
for all state in Kubernetes -
kubectl, by comparison, serves as
a convenient CLI for interacting with the Kubernetes API.
Kubernetes Cluster State in
Kubernetes was built around the idea that all components are stateless
and largely function by asking the API server for the desired state,
comparing that to the current state, making changes, and saving the
results back to
etcd via the Kubernetes API. If one of these
components was restarted, it would pick up where it left of by starting
its cycle of interaction with the Kubernetes API over again.
etcd seems to have been chosen because it is a
distributed reliable key-value store.
I believe I read somewhere that we may see this portion of Kubernetes
become “pluggable” in the future but for now
etcd is all there is.
One very important thing to know is that only the Kubernetes API
can talk directly to
etcd - everything else only knows about the API
Priming the Environment
Download and install Vagrant if you have not already. You may also need to install Virtual Box if you do not have it already.
From there, open a terminal and provision your Vagrant box with the following commands:
Update: Later in this post, I ran into an issue with
ubuntu/artful64related to disk space being constrained to ~2GB instead of the expected ~10GB. It’s possible that when you are reading this has been addressed, but incase you haven’t there is a workaround for the bug at the appropriate point in the article.
From there, you will
ssh into the Vagrant box and install
Docker using the instructions
on the Ubuntu page
(also copied below). In the instructions the
sudo apt-get update will
probably take a few minutes to complete but ensures you are installing
the latest packages. Also you will probably get asked about using
additional disk at a few of these steps and I answered
Y as I went
through this. This is similar to the previous article, but
added to help with some YAML to JSON conversion.
Docker should have said “hi” and you should be ready for the next step.
Then we need to grab
kubelet - later it will instruct Docker to run
containers for us.
At this point we are ready to start tackling setting up
Firing up the API Server
Now we pick back up where Kamal’s post
jumps in with “Starting the API server”. As he said, we first need to get
etcd running. We create a directory for storing the state and getting
it running with Docker.
One thing to note is that we are using
--net=host so that the API server
can talk to
127.0.0.1 on the default port
And then we grab the Kubernetes API server binary:
With the binary in place with the correct permissions, we can fire it up.
(Makes me remember The Crow to say
that, but’s probably not the best imagery here. ;) ) We need to tell
kube-apiserver where to find
etcd and the IP range of the cluster.
10.0.0.0/16 will work for us but longer term you may want to better
understand what this choice implies.
Permission denied?! This next piece will make sense only in the context
of this VM - rerun that last command with
sudo. You will want to be
very cautious of running processes under
sudo in any other environment.
You should see something like the following and the prompt will not return while this server is running.
In another terminal, we need to
ssh back into the Vagrant box and use
curl to check the API server out.
That gave us a list of nodes running in the cluster. It’s empty because we haven’t set any up yet.
Kamal points out that
resourceVersion is used for concurrency control.
This allows a client to send back changes to a resource along with a
referecnce to the version it was at before the change so the server can
determine if there was a conflicting write since the client last
requested this information.
Kamal also makes use of the
jq utility to parse JSON from these sorts
of responses. For this part, we only want to see the
items so we can
try the following and see that we also have no pods currently.
Our API server is running and now it’s time to add a node.
Adding a Node
In the previous post, we
kubelet watching a directory for pod manifests. This
time we want to add pods via the API server. To do that we need to
kubelet pointing to the API server.
In order for
kubelet to find the API server we need to tell it where
to look using a
kubeconfig file. For our purposes create a file in
the current directory called
kubeconfig using the link provided.
kubelet command will need to be run with
sudo in the
Vagrant box for ease of learning but is not recommended in other
The above command is one of the departures from Kamal’s post. In
that, he used
--api-servers (instead of
--kubeconfig) which is no
When we operate with
kubelet talking to an API server it opens up
some interesting opportunities - for one, we can add another node
to the cluster pointed at the API server and not need to reconfigure
the API server to make use of it.
To verify that
kubelet knows about our API server we can open up
(yet) another terminal session and ask about node(s).
Now that we have a running node, let’s add a pod.
Adding a Pod
Much like in the previous post,
we want to get a simple pod definition in place to test this out. We will
reuse much of the prior definition with an important change - we need to
specify where the pod should run. This is not needed in a complete
Kubernetes cluster - because there is a component called
kube-scheduler to “schedule” pods on nodes - but since we are building
this up one component at a time we need to take on a little more
responsibility in our pod definition.
The important difference from the definition in part 1 is adding
nodeName: ubuntu-artful where the value is the name of the node as
found in the prior
Another inconvenience here is that while a more complete Kubernetes
cluster can be sent commands via
kubectl using YAML files as input,
kube-apiserver only understands JSON. Kamal offers up the following
as a way to convert our YAML file appropriately.
Now we can submit the request to the API server to add the pod using
Kamal’s example. Make sure that the API server knows that the file
we are sending is JSON using the
After a few moments, we should be able to see that the pod is now running on our node.
OK, so now we are learning something else - we need to be sure that we have enough disk available for our pods.
Brief Side-Trip into Vagrant Image Disk Size
Checking the current disk we see the following:
After a bit of Googling, I found that this is
a known issue
ubuntu/artful64 Vagrant box that we are using. While it will
be nice when the underlying bug is addressed (it wasn’t as of 2018-02-25),
we can get around the problem for now with the following:
Back to Provisioning Our Pod
If we check the pod status again at this point, we will see that it is still marked as failed.
If we try to
POST the pod again we will get a rejection like the
Here again, we find something that would be less of an issue in a more
complete Kubernetes cluster. The pod isn’t getting rescheduled now
that we have more space because we have to do that ourselves - we
kube-scheduler to sort that out for us.
That command tells the API server to remove the pod called
can confirm it is gone with the following:
Hopefully, you will see what I saw here which is nothing. Now we can resubmit the pod definition and verify it went as expected.
The pod is now running and assigned the IP
172.17.0.2 by Docker.
Kamal references this
as a good place to start learning about Docker’s network configuration.
Before we move on, we should double check that
nginx is actually
listening on that IP:
Kubernetes CLI -
Now that we have learned how to interact with the Kubernetes API
server directly we can take a step up in terms of abstractions.
You could use
jq and manually convert YAML to JSON -
or you could use the great CLI that Kubernetes offers -
Much like Kamal said, I think you will like it.
To get started, let’s fetch the client:
Once we have this, getting a list of nodes (or pods) is pretty simple:
Again, Kamal was right - this is much easier to type and easier to read
the responses. To test this out we will spin up a copy of the existing
nginx pod using
kubectl. Start by duplicating the pod manifest YAML
file and replace the name.
Then we can use
kubectl create to start that second copy.
Depending on how quickly you execute that second command, you may find
that some of the containers are still being allocated. Give it a moment
./kubectl get pods again until you see what I have above.
If we want to get more information about a pod, we can use
And once again, we can double check that this second copy of
serving requests at the IP listed above.
At this point, we have a sense of what it means to start a pod in Kubernetes from the command line. There are a number of additional components you would find in a more complete Kubernetes cluster but it helps to take it a piece at a time. If nothing else, we can better appreciate how each new component makes our lives simpler by removing work we previously had to do ourselves.
Before we leave this, we should look at how to clean up our running
pods. I think you will find this a bit easier than using
A last cleanup step is to
exit from the Vagrant box and
Next we will look at
kube-scheduler in a
future post so we no longer
need to specify the node that our pod runs on.