The gap between Minikube and Kubernetes in production is vast and full of pain. Want to know what it’s like to implement Kubernetes (k8s) in production for the first time before you collect the bumps and bruises yourself? Join me to learn about the ups and downs of one company’s initial experience with k8s - including who was involved, the scope, and what was intentionally deferred.
We will cover:
- Why even consider Kubernetes (Hint: you might not benefit from it!)
- How a real company approached their first proof-of-concept
- How goals changed during implementation of k8s
- What we learned so far
- What we still need to learn
Background
In 2017, the company I worked for, Infinite Campus, was underway using containers for some application deployments. Notably a relatively new Node app where containerized deployment helped us freeze the npm dependencies in development and preserve them through production no matter what happens on the internet.
With more containerized deployments on the horizon we began looking to level-up on how to manage containers in production. While we could continue to orchestrate Docker container deployments with Chef — our configuration management tool of choice — we felt we would end up building tooling others had already built. And moving beyond a few nodes, even Docker-Compose wouldn’t be helpful enough.
When Do I Need Kubernetes
It’s not for everyone. There is a lot to learn and complexity to manage. If your needs don’t require that level of complexity or scale, you would likely be better off sticking with something much more straight-forward like Docker-Compose.
A Possible Heuristic
You can find advice around the Internet for when it does or does not make sense to use Kubernetes or other multi-host container orchestration. Container Solutions offers up a 6 step approach and insists that k8s is necessarily the last on the list. From what I have seen personally, their points seem valid.
In addition to keeping list like theirs in mind, I offer the following approach: If you don’t have 10+ production VMs you are managing across 5+ applications each spread across multiple hosts, you could probably get by with something less complex. The overhead of adapting your development to Kubernetes should be understood before committing to it. The additional considerations of operating Kubernetes itself is something to keep in mind as well.
If you want to use containers in your deployments beneath these (made up) thresholds consider Docker-Compose and configuration management like Chef or Ansible to care and feed for the hosts.
Why Kubernetes, Why Not Swarm
By late 2017, it was already becoming clear that Kubernetes was “winning” over Docker Swarm in the hearts-and-minds department. Even Docker seemed to be embracing k8s. Since we were going to make a large technology learning invesment, Kubernetes seemed the best bet in an evolving landscape.
How Hard is K8s to Learn
It depends. ;)
Getting started with Kubernetes basics in late 2017 wasn’t super polished. There were more and more posts coming out where people were giving tutorials. Minikube could get you started learning the concepts on your dev box. And Kelsey Hightower’s popular Kubernetes the Hard Way gave you a deep dive into how to bootstrap a cluster including creating and distributing certificates.
Even so, there are a lot of concepts to grok. Thankfully, there were a few internet gems that helped.
Enter: Good Visual Overviews
When learning Kubernetes, I found that good visual overviews are incredibly helpful. Kubernetes has so many moving pieces that it is difficult to keep track of them all — especially if you try to get them all into your head at once.
Don’t take the title of the first as dismissive, much like the awesome “Git For Ages 4 And Up”, it addresses the complex topic of container orchestration in an approachable way. I highly recommend it as a way of getting a visual model of some of the basic Kubernetes layers including containers, pods, labels, replication controllers, services, and volumes.
The Google comic takes a different tact — it comes at the “what” and “why” of Kubernetes from a sysadmin point-of-view. It also has a bit of silliness mixed in — and a small pitch for Google Kubernetes Engine (GKE) — but does a good job of making its case a piece at a time.
But the Magic, How Does it Work
If you are like me, and you are hoping to use Kubernetes in production, you are uncomfortable with magic in your platforms — and Kubernetes has a lot of magic.
To get your head around those piece you can build up a Kubernetes cluster one component at a time. I wrote a series of posts on the topic based on my exploration of the API and control plane during the 2017-2018 winter holiday.
More recently, there are good some really good hands-on tutorials over at Katacoda that make a lot of sense to work with beyond whatever you can learn from Minikube.
How Deep is This Well
More than likely, once you have convinced yourself you know how to do deployments with Minikube on a single node and you have your head around the API and control plane, you quickly find yourself needing to level-up on a number of topics like the many layers of networking, security/secrets, and the meaningful differences between how you host Kubernetes and the resulting features that are — or are not — available.
For us, we decided to set out on a proof-of-concept (PoC) with just enough scope to learn a lot but attempting to avoid completely going off the deep end. We knew that if we didn’t treat this as production, we would have too many opportunities to cut corners. On the other hand, this wasn’t the time to move our company from our data centers into the public clouds and learn that infrastructure as well. In addition, hosted Kubernetes was — and still is — rapidly evolving. By focusing on bare k8s, we hoped our knowledge would be transferrable to whatever infrastructure we went with next.
Who Was Involved
As important as scope is considering who will be involved in your first serious attempt at Kubernetes. By “serious”, I mean production quality and not “works on my box”. ;)
For us, we were able to get some cross-functional participation from the beginning — and I think this is critical. Personally, I came at Kubernetes from a software engineering background and not operations. From my own explorations to that point, I knew that having serious help with networking and other “ops” skills would be critical.
We had the following people/skill combinations participating more-or-less full-time at the beginning:
- Software Engineering (2): A “junior”, but talented, individual and myself (15+ years experience in a number of applications)
- Operations (2): A well-rounded ops engineer with networking expertise and one very focused on production application support, both with a fair amount of experience
We also had folks helping in more supportive capacity and time-commitment:
- Software Engineering (1): Another very experienced (20+ years) software engineering manager with a passion for security
- Project Manager (1): Experienced in many projects and able to help provide some structure for our work
Later, the team would change — generally shrinking in terms of availability — leading to challenges I will cover in a bit. In the beginning this felt like a pretty good group with support from their supervisors to participate in helping the company prove this out.
Our Proof-of-Concept Scope
When working with Kubernetes for the first time, it is important to consider what is enough scope to learn something valuable while not biting off more than you can chew in your initial outing.
We considered at least the following:
- Production Stability and Repeatability
- Existing Containerized Applications
- Existing Infrastructure and OS
- Existing Storage Technology
Production Stability and Repeatability
As I said, we wanted our PoC to be “production” — meaning we wanted production-like guarantees of service and repeatability in our approach. However, we also weren’t ready to risk our customers — or our revenue — while we learned.
Existing Containerized Applications
Since my Data Science team was experiencing some ops issues already and needed to replace our internal ETL (Airflow) and analytics (Metabase) VM’s with something more stable than their initial “alpha” installs, I volunteered to take this on with the goal of getting production-level service for our internal users. We needed to be able to depend on these systems for important functions but it’s easier to apologize in person and the existing systems set a low bar of service to beat. ;-)
As importantly, both Airflow and Metabase appeared to be in a good place with already available container images — we didn’t want to have the added complexity of breaking new ground on this part. (Note: This came back to bite us later when we realized that good container images doesn’t necessarily mean that the community had solved how to make them work well in Kubernetes.)
I will also admit at this point that I significantly underestimated just how much learning we would need to do and the time it would take to do so. Was it valuable for the team? Sorta. Was it valuable for the company to start this learning in earnest? Definitely.
Existing Infrastructure and OS
We went with CentOS VMs because we knew we could provision them easily. We were concerned that learning a container operating system like CoreOS would be too much to take on this first outing and we were itching to get the first pass up and running. (Note: This burned us later when CentOS didn’t support some features like LDAP that we wish we had better access to for user provisioning and role-based access controls (RBAC).)
As I said before, learning how to operate effectively in public cloud infrastructure also felt like taking on too much — espeacially when we could get more than enough VMs in our existing data centers without many questions.
Existing Storage Technology
As part of working with Airflow, we knew we would need to consider storage in a couple ways. We needed a database for Airflow’s metadata — for us that was Postgres. We also knew that Airflow would require all pods running the Airflow container to be synchronized to the same code and that code was the most likely thing to change and therefore not included in the container image.
We knew NFS and chose to shy away from gluster or other better-suited storage technologies. We were eager to learn them but needed to get something working.
How Goals Changed During Implementation of K8s
As we progressed through this internal production PoC, we revisited a few of our goals and approaches. Some of this was due to our growing knowledge of what container approaches were already proven. Some was due to updates coming out between when we planned out the effort and when we were ready for that portion.
HA Postgres with Stolon
When we started planning, we decided we would take the leap and try putting a database into Kubernetes. Many on the Internet were actively debating whether or not this was a good idea. The tweet below from Kelsey Hightower is one example:
Kubernetes supports stateful workloads; I don't.
— Kelsey Hightower (@kelseyhightower) February 13, 2018
Even so, this seemed to us a thing that we could try since the database we were looking to manage in k8s wasn’t particularly large or performance-sensitive.
From having decided to use a community container image for Postgres, we explored StatefulSets and how to share that state across nodes using NFS mounts on the hosts.
As we were implementing, one of the team found an interesting Postgres-related project called Stolon. This project had a Kubernetes-friendly architecture and would allow us to not only learn about managing a database in k8s but also give us our first taste of High Availability Postgres.
Our scope was creeping. ;-)
NFS as a Service
We also evolved our use of NFS storage in the Kubernetes cluster. At first we were attaching the same NFS volume to each of the hosts to allow for simplistic shared state. The primary reason we wanted this was the expectation that we would need to update DAG source code for Airflow into all its pods at once — and didn’t need to restart Airflow to make this happen. But we also made use of this shared storage for Postgres to allow it to recover (before we were using Stolon for HA) on any node.
As we began to work with NFS on the nodes, we realized that we could also mount NFS as a service in the cluster and carve out volumes for the pods from that rather than on the hosts themselves. Life got simpler at the host level and it felt quite a bit more “Kubernetes”-ish.
Something we didn’t do — and it caused some headaches — was to stop using NFS for the volumes Postgres was writing to when we moved over to HA via Stolon. So rather than letting only the Postgres application manage the syncing between nodes we were also addressing the syncing with NFS. We considered swapping back to ephemeral storage on the hosts but left it alone for the time being since things seemed to be working albeit less efficiently than we thought it could be.
Scaling Airflow
When we started the effort we had not been planning to run Airflow in a scaled mode. In other words, we were not planning to have multiple workers initally. We knew it was possible but this felt like a complexity we could defer.
As we were also getting used to Airflow outside of Kubernetes we found that using the LocalExecutor had some limitations that the CeleryExecutor — the one that allows multiple workers — did not have. So in our Docker-Compose instance of Airflow we switched over and found that the complexity wasn’t too bad. With that knowledge we took another look at how Airflow would be best deployed in k8s with an eye toward scaling the workers independently.
It was around this time that we learned that while Airflow was in a good place from a container point of view, it was still immature in how it worked in Kubernetes. This was during Airflow’s v1.9 stage. It did not have community Helm charts yet let alone the KubernetesExecutor or Operator it would pick up in v1.10 that released in the latter half of 2018. Because of this lack of maturity we found ourselves learning alongside the Airflow community what concessions and considerations were necessary for k8s harmony.
Open source software moves quickly. Keep that in mind as you begin your journey. For us, major changes in both core Kubernetes and Airflow were released in the time we were trying to get our feet under us. It was exciting — but also a moving target.
What We Learned So Far
We had a number of take-aways during our implementation of Kubernetes.
Don’t Roll Your Own
There were a number of pain points we ran into that were incredible learning opportunities about Kubernetes but mainly were challenges to spinning up a functioning cluster. Since we had no intention of selling k8s as a service — way outside our core business — we would benefit by saving precious employee time in trade for licensing someone else’s expertise.
Certificates are Important
We knew we still had a fairly immature certificate generation and distribution infrastructure going into this effort. What we found was that you really need to have something solid in place to keep k8s humming.
As referenced in Kelsey Hightower’s “Kubernetes the Hard Way”, you need to “generate TLS certificates for the following components: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy.” This includes certificates for each node and all of the controllers. All of that is before you get around to individual applications and how you want to handle external SSL connections to your services.
If you don’t have a convenient certificate authority for this — or much experience handling this repeatably and automatically — you will find you have some leveling up to do.
Networking Knowledge is Critical
I mentioned early on that we had one member of the core team that was a skilled “Ops” person with a deep networking background. That turned out to be even more important than originally thought.
As explained in this post by Kirill Goltsman of Supergiant:
“Kubernetes cluster networking is perhaps one of the most complex components of the Kubernetes infrastructure because it involves so many layers and parts (e.g., container-to-container networking, Pod networking, services, ingress, load balancers), and many users are struggling to make sense of it all.”
Navigating through those layers — especially when things don’t go as planned — is painful. Early on in the effort we were able to learn from our local expert and power through many of these challenges. Later this became a bigger challenge when that same expert was pulled off onto other work leading to significant delays in making progress.
Secrets aren’t Secret
As we got involved with Kubernetes we had a limited understanding of how it managed secrets — think connection strings and credentials that pods need to run. We were hearing competing ideas about how to inject secrets into containers — should you use environment variables or volume mounts? — and found we needed to learn a lot more about how each approach plays well or not in k8s.
One of our initial surprises was how Kubernetes stores secrets internally — base64 encoded. That means that an administrator of a k8s cluster would have relatively straight-forward access via the API to all the “secrets” stored in the cluster.
On a more positive note, we learned about “sealed
secrets” via kubeseal which allow you to properly
encrypt secrets to be stored in source control of
your Kubernetes deployment files. Even so, while
they are encrypted nicely in source control they
are still decrypted and stored base64 encoded in
the cluster.
More promising yet would be integration with an external secret store like HashiCorp Vault.
Some Applications Won’t Benefit Much
I mentioned earlier that there was a fair amount of change that came with how we could best use Airflow in Kubernetes. We had related learnings for Metabase as well.
Metabase is an application that provides a consistent UI to visualize data from many underlying data sources. But as it does this, it does not take on much load directly. Most of the performance issues come from running queries — often not well optimized — against those data sources. As a result the Metabase application doesn’t really benefit from horizontal scaling.
But that’s not actually the kicker.
We thought, sure, it might not benefit performance-wise but perhaps there is something to be gained from having a few instances running for high availability. Turns out that Metabase is not “cluster aware”. In other words, multiple instances running against the same metadata backend will act independently of each other. For example, the “Pulses” feature which sends scheduled reports will send duplicates as each instance fires on that schedule.
So what we got from running Metabase in Kubernetes was the ease of ensuring that a copy of the application was running and the ability to quickly deploy a new version when we wanted to. However, due to the application state that needed to be migrated with some upgrades we did find that we needed to consider more complex rollback strategies that I won’t go into here.
What Still Needs to be Learned
We covered a lot of ground in our first serious outing with Kubernetes. Even so, there was a lot either kept out of the scope initially or that we learned existed during the process and never got to.
CI/CD Deployment into Kubernetes
We thought quite a bit about how we would do more effective deployments through CI/CD into Kubernetes. We considered how that would change — or not — some of the decisions we had made around how containers were implemented.
One example was the decision to mount the Airflow DAG code into the pods rather than building containers with that code baked in. Very likely we should be using multi-stage Docker builds to keep a base image of the Airflow application and layer on the more volitile DAG definitions. In this way we avoid needing to syncronize code between the Airflow pods and significantly simplify the deployment from a CI/CD pipeline.
CoreOS and/or Public Cloud
We intentionally deferred some significant operations learning around new operating systems or how to usefully maintain Kubernetes nodes in a public cloud. Follow up activities would need to consider how the k8s nodes were provided and operated in a more complete way. All of this has evolved significantly since the early days of the PoC at the beginning of 2018 — and there is very little sign of slowing. A big decision is where will be the next “stake in the ground” around Kubernetes hosting.
Improved Storage Options and External State
There are a few really solid storage technologies that are
much better suited to Kubernetes like gluster and cephs.
Looking into either would be an improvement for volume
mounting into pods.
Another consideration would be whether maintaining state within the cluster is even the right choice. Just because you can doesn’t mean you should — the payoff isn’t always there. Having your data stores managed external to your cluster allows for specialized expertise to come into play and is likely a better choice if you have heavy performance or scaling needs.
Namespaces and Deployment Environments
During the time we spent digging into Kubernetes, we debated a number of different approaches to mimicking the deployment environments that we were used to. E.g. Dev, Integration, Production. You can certainly have more but for simplicity lets just consider those.
For Dev, is that really Minikube on a developer machine? There are some scenarios where this doesn’t really help you understand how your code will really work — especially if you are working on something more performance sensitive.
For separating Integration and Production, should that be done using namespaces or separate clusters? If you go with namespaces, you can still run into issues where testing can consume resources you really need to keep safe for Production. If you separate out with different clusters, you start to lose the benefit of clustering in the first place for managing resources more effectively.
And those considerations are just for applications. It gets more complex when you realize you need to test new versions of Kubernetes as well and every three months there is a chance for a major change that forces you to reconsider how your deployments work!
In Closing
Think Before You K8s
Kubernetes brings a lot to the table for simplifying the orchestration of containerized applications/services across multuple nodes. That said, it does not come for free. There are development and operational considerations and costs to consider in terms of complexity.
Be sure you have problems Kubernetes is meant to solve before committing your team’s precious time to learning to use it in production.
Plan to Learn — In Stages, Over Time
There is a lot to learn when aspiring to operate Kubernetes in production. Don’t try to tackle all of it at once and don’t expect that it will be a weekend project — or even one sprint, if that’s how you roll.
In addition, the ecosystem is rapidly evolvling and breaking changes
are likely to come in some form nearly every quarter in Kubernetes
“minor” releases (e.g. 1.x, where x supposedly means additive).
You will want to be sure you can dedicate sufficient time to keeping
up with how that ecosystem is changing and what you will need to —
or want to — change as a result.
People Matter — Especially Network Folk
As with many things in software development, Kubernetes is a team sport. You need a fairly broad set of skills in your team to make meaningful progress — especially at the beginning. Make sure you have a team of people whith sufficient time to dedicate to to the work and who will communicate well with each other.
And having a networking expert in the mix will be a massive help!
Expect Plans to Evolove
As I said before, the lanscape of Kubernetes is changing rapidly. Along with it are the various projects that are learning to adapt to that new current state of k8s. You may find yourself needing to make due until a new capability is released in the platform or the application you are planning to migrate to Kubernetes. Or entirely revisit your approach as you learn about a better way.
Good Luck
I hope sharing this journey is helpful as you debate — or plan — taking on Kubernetes in your own team. Be patient with yourself and your team as you learn.
And don’t feel like a lesser developer/operator if you choose to forgo Kubernetes — not everyone will benefit.
For those looking for a visual accompaniment to this post, I put the slides my 2019 Minnebar presentation up on SpeakerDeck.