PythOps

kubernetes the hard way part 1

Last update: 23 December 2019

Introduction

Even though there are tools out there that automate the creation of k8s cluster, they aren't that flexible when it comes to customization.
So if you are curious about k8s and want to know how to create your own cluster like a pro, you're in the right place !
This is part 1 of a series of posts where I'll be explaining how to deploy a highly available cluster.

At the end of this post, you'll be able to deploy a highly available k8s masters as shown below.

So let's get started !

Prerequisites

Before starting you need to have these tools installed in your workstation.

Once you install these tools, the next step is to clone the Ansible playbooks from GitHub

$ git clone https://github.com/pythops/k8s_the_hard_way
$ cd k8s_the_hard_way


Setup the lab

We're gonna use Vagrant to create the VMs.

$ vagrant up

You should have 4 VMs up and running

$ vagrant status
Current machine states:
lb                        running (virtualbox)
master1                   running (virtualbox)
master2                   running (virtualbox)
master3                   running (virtualbox)


Deploy the Etcd cluster

Etcd is a key-value store that is used by k8s to store all the cluster data.
We're gonna deploy a highly available cluster with 3 nodes. With this setup we can afford to lose up to 1 node and still having a working cluster.

PKI certificates

We're gonna create a set of certificates to secure all the communication.

Create the PKI certificates
$ ansible-playbook k8s-pki.yaml -t etcd

This will create and store all the certificates inside the directory pki/etcd

pki
├── etcd
│   ├── etcd-peer-ca.crt
│   ├── etcd-peer-ca.csr
│   . . . 
│   └──  k8s-etcd-3-server.key

Deploy Etcd cluster

$ ansible-playbook etcd.yaml

Verification

$ vagrant ssh master3 -c "ETCD_VER=v3.3.18 etcdctl --cert-file /etc/etcd/k8s-etcd-3-server.crt --key-file /etc/etcd/k8s-etcd-3-server.key --ca-file /etc/etcd/etcd-server-ca.crt cluster-health"
member 2cba54b8e3ba988a is healthy: got healthy result from https://10.0.0.103:2379
member 7c12135a398849e3 is healthy: got healthy result from https://10.0.0.102:2379
member f2fd0c12369e0d75 is healthy: got healthy result from https://10.0.0.101:2379
cluster is healthy

Now that we have an etcd cluster up and running, let's move to the next step and deploy the k8s masters.

Deploy the k8s masters

A k8s master has 4 components:

For this setup we're gonna deploy only kube-apiserver, kube-controller-manager and kube-controller-manager

PKI Certificates

We're gonna create a set of certificates to secure the communications within the master components and with the outside world as well.

For more infotmation about the k8s PKI, check the k8s official doc here: https://kubernetes.io/docs/setup/best-practices/certificates/

Create the PKI certificates
$ ansible-playbook k8s-pki.yaml -t master

this will create all the certificates in pki/k8s directory

├── k8s
│   ├── k8s-ca.crt
│   ├── k8s-ca.key
│   ....
│   └── service-account.key

Kubeconfig files

kubeconfig files are used to access the cluster. They are used by any component that needs to talk to the apiserver.

Create kubeconfig files
$ ansible-playbook kubeconfig.yaml -t master

This will create all the kubeconfig files in the kubeconfig directory

kubeconfig
├── kube-controller-manager.kubeconfig
└── kube-scheduler.kubeconfig

Download the k8s binaries

To save time and bandwidth, we're gonna download the binaries to the host machine and then copy them to the VMs

$ ansible-playbook k8s-download.yaml

This will download the k8s master components to a bin directory

bin
├── kube-apiserver
├── kube-controller-manager
└── kube-scheduler

Deploy Master components

Deploy master components

$ ansible-playbook k8s-master.yaml

One important option in apiserver options is --encryption-provider-config which encrypts all the data before storing it in the etcd. the encryption key is defined in roles/k8s-master/defaults/main.yaml

Cluster admin user

PKI certificates

$ ansible-playbook k8s-pki.yaml -t admin

This will create the admin certificate and private-key in pki/user-accounts directory

pki
└── user-accounts
    ├── admin.crt
    ├── admin.csr
    └── admin.key

kubeconfig

We need to generate the kubeconfig file as well

$ ansible-playbook kubeconfig.yaml -t admin
kubeconfig
├── admin.kubeconfig


Deploy the Load Balancer

We're gonna deploy a TCP loadbalancer using Nginx

$ ansible-playbook loadbalancer.yaml


Verification

Let's verify now that our setup is working

$ kubectl --kubeconfig kubeconfig/admin.kubeconfig get componentstatus
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"} 

Congratulations 🎉

So far we have deployed a highly available k8s masters. Let's move to part 2 to deploy k8s nodes.

Recommended reading