kubernetes the hard way part 1

Last update: 23 December 2019


Even though there are tools out there that automate the creation of k8s cluster, they aren't that flexible when it comes to customization. So if you are curious about k8s and want to know how to create your own cluster like a pro, you're in the right place ! This is part 1 of a series of posts where I'll be explaining how to deploy a highly available cluster.

At the end of this post, you'll be able to deploy a highly available k8s masters as shown below.

So let's get started !


Before starting you need to have these tools installed in your workstation.

Once you install these tools, the next step is to clone the Ansible playbooks from GitHub

$ git clone
$ cd k8s_the_hard_way

Setup the lab

We're gonna use Vagrant to create the VMs.

$ vagrant up

You should have 4 VMs up and running

$ vagrant status
Current machine states:
lb                        running (virtualbox)
master1                   running (virtualbox)
master2                   running (virtualbox)
master3                   running (virtualbox)

Deploy the Etcd cluster

Etcd is a key-value store that is used by k8s to store all the cluster data. We're gonna deploy a highly available cluster with 3 nodes. With this setup we can afford to lose up to 1 node and still having a working cluster.

PKI certificates

We're gonna create a set of certificates to secure all the communication. * k8s-etcd-{1..3}-peer.crt are used to secure the communication between the etcd nodes. * k8s-etcd-{1..3}-server.crt are used to secure the communication between the cluster and the outside world.

Create the PKI certificates
$ ansible-playbook k8s-pki.yaml -t etcd

This will create and store all the certificates inside the directory pki/etcd

├── etcd
│   ├── etcd-peer-ca.crt
│   ├── etcd-peer-ca.csr
│   . . .
│   └──  k8s-etcd-3-server.key

Deploy Etcd cluster

$ ansible-playbook etcd.yaml


$ vagrant ssh master3 -c "ETCD_VER=v3.3.18 etcdctl --cert-file /etc/etcd/k8s-etcd-3-server.crt --key-file /etc/etcd/k8s-etcd-3-server.key --ca-file /etc/etcd/etcd-server-ca.crt cluster-health"
member 2cba54b8e3ba988a is healthy: got healthy result from
member 7c12135a398849e3 is healthy: got healthy result from
member f2fd0c12369e0d75 is healthy: got healthy result from
cluster is healthy

Now that we have an etcd cluster up and running, let's move to the next step and deploy the k8s masters.

Deploy the k8s masters

A k8s master has 4 components: * kube-apiserver: validates and configures data for the api objects( pods, services etc) * kube-controller-manager: watches the state of the cluster through the apiserver and makes changes to move the current state towards the desired state * kube-scheduler: finds the best node for newly crated Pods to run on. * cloud-controller-manager (optional): embeds cloud-specific control loops.

For this setup we're gonna deploy only kube-apiserver, kube-controller-manager and kube-controller-manager

PKI Certificates

We're gonna create a set of certificates to secure the communications within the master components and with the outside world as well.

For more infotmation about the k8s PKI, check the k8s official doc here:

Create the PKI certificates
$ ansible-playbook k8s-pki.yaml -t master

this will create all the certificates in pki/k8s directory

├── k8s
│   ├── k8s-ca.crt
│   ├── k8s-ca.key
│   ....
│   └── service-account.key

Kubeconfig files

kubeconfig files are used to access the cluster. They are used by any component that needs to talk to the apiserver.

Create kubeconfig files
$ ansible-playbook kubeconfig.yaml -t master

This will create all the kubeconfig files in the kubeconfig directory

├── kube-controller-manager.kubeconfig
└── kube-scheduler.kubeconfig

Download the k8s binaries

To save time and bandwidth, we're gonna download the binaries to the host machine and then copy them to the VMs

$ ansible-playbook k8s-download.yaml

This will download the k8s master components to a bin directory

├── kube-apiserver
├── kube-controller-manager
└── kube-scheduler

Deploy Master components

Deploy master components

$ ansible-playbook k8s-master.yaml

One important option in apiserver options is --encryption-provider-config which encrypts all the data before storing it in the etcd. the encryption key is defined in roles/k8s-master/defaults/main.yaml

Cluster admin user

PKI certificates

$ ansible-playbook k8s-pki.yaml -t admin

This will create the admin certificate and private-key in pki/user-accounts directory

└── user-accounts
    ├── admin.crt
    ├── admin.csr
    └── admin.key


We need to generate the kubeconfig file as well

$ ansible-playbook kubeconfig.yaml -t admin
├── admin.kubeconfig

Deploy the Load Balancer

We're gonna deploy a TCP loadbalancer using Nginx

$ ansible-playbook loadbalancer.yaml


Let's verify now that our setup is working

$ kubectl --kubeconfig kubeconfig/admin.kubeconfig get componentstatus
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-2               Healthy   {"health":"true"}
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}

Congratulations 🎉

So far we have deployed a highly available k8s masters. Let's move to part 2 to deploy k8s nodes.