Hands on GitOps with FluxCD

Last update: 01 January 2022

Introduction

GitOps is a set of practices where the source of truth for your applications and/or infrastructure is managed by git.

The two major benefits of implementing GitOps in my opinion are:

Any manual changes to your platform are detected and overriden to match the expected state defined in git.
It makes deploying complex platforms very easy. Think about a microservice platform with hundreds of microservices that you want to replicate elsewhere.

In this blog post, we're going to use FluxCD to implement GitOps for application management.

Let's start !

Requirements

We're going to need:

- Flux CLI that you can download here https://fluxcd.io/docs/cmd/

- k8s cluster. For this demo, we're going to use minikube

Demo

We're going through a typical workflow of an app, from building a simple web app to deploying it inside Kubernetes.

Demo app

I've created a simple web app using Flask that I hosted in this repo https://github.com/pythops/gitops-demo-app

├── app.py
├── Dockerfile
├── kustomize/
├── Makefile
├── Readme.md
└── tests/

To deploy the app in Kubernetes, we are going to use kustomize. The kustomize manifests can be found in the kustomize folder in the root directory

├── kustomize
│  ├── deployment.yaml
│  ├── kustomization.yaml
│  └── service.yaml

One important thing to notice here is the image tag.

#File: kustomize/deployment.yaml
   ...
   image: pythops/gitops:master-a8158c7c-1640370014 # {"$imagepolicy": "flux-system:image-repo-policy"}
   ...

Initially I added an arbitrary tag (check the initial commit here)

#File: kustomize/deployment.yaml
   ...
   image: pythops/gitops:latest # {"$imagepolicy": "flux-system:image-repo-policy"}
   ...

But then let Flux patches the tag whenever a new image is built and pushed to the image registry. Later we will see how can we achieve this.

The CI with Github actions

We are going to use Github actions that will run some tests, build the container image and push it to Dockerhub.

The important thing to Notice here is that the images tag are in this format <branch-name>-<hash>-<timestamp>

This format is important, as we're going to find it later in the Flux configuration.

#File: .github/workflows/ci.yaml
      ...
      - name: Generate build ID
        id: prep
        run: |
          branch=${GITHUB_REF##*/}
          sha=${GITHUB_SHA::8}
          ts=$(date +%s)
          echo "::set-output name=BUILD_ID::${branch}-${sha}-${ts}"

      ...

      - name: Build and publish container image
        uses: docker/build-push-action@v2
        with:
          push: true
          context: .
          file: ./Dockerfile
          tags: |
            pythops/gitops:${{ steps.prep.outputs.BUILD_ID }}

The CD with Flux

Install Flux inside k8s

Flux needs a place where to store its installation manifests. For that purpose, I'm going to use Github.

First, we're going to generate a token so Flux can connect to Github

$ export GITHUB_TOKEN=<Github token>

Run Flux pre checks

$ flux check --pre
► checking prerequisites
✔ Kubernetes 1.21.6 >=1.19.0-0
✔ prerequisites checks passed

Then deploy Flux

$ flux bootstrap github \
  --owner=pythops \
  --repository=gitops-demo-flux \
  --path=demo \
  --components=source-controller,kustomize-controller \
  --components-extra=image-reflector-controller,image-automation-controller \
  --private=false \
  --personal=true

...
► confirming components are healthy
✔ image-automation-controller: deployment ready
✔ image-reflector-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

This command will:

Create a public Github repo https://github.com/pythops/gitops-demo-flux
Deploy Flux components inside minikube

To verify that all the Flux components are running correctly

$ kubectl -n flux-system get pods
NAME                                           READY   STATUS    RESTARTS   AGE
image-automation-controller-7b4fb699f4-xqp98   1/1     Running   0          37s
image-reflector-controller-65fb48496c-gnmmv    1/1     Running   0          37s
kustomize-controller-655b6bb4d5-v5hsb          1/1     Running   0          37s
source-controller-9df84c994-dpxnv              1/1     Running   0          37s

Finally, we need to generate a deploy key so Flux can commit to the gitops-demo-app repo. The public key should be added to Gihub.

$ ssh-keygen -q -N "" -f ./identity
$ ssh-keyscan github.com > ./known_hosts

$ kubectl -n flux-system create secret generic ssh-credentials \
    --from-file=./identity \
    --from-file=./identity.pub \
    --from-file=./known_hosts

Configure the app

We're going to clone the repo gitops-demo-flux locally

$ git clone https://github.com/pythops/gitops-demo-flux

This initial structure and files are created when I bootstrapped Flux

└── demo
   └── flux-system
      ├── gotk-components.yaml
      ├── gotk-sync.yaml
      └── kustomization.yaml

To configure the management of the app gitops-web-app by Flux, we're going to create the following yaml file and commit it to the repository inside the demo directory.

#File: gitops-demo-app.yaml

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: gitops-demo-app
  namespace: flux-system
spec:
  interval: 30s
  url: ssh://git@github.com/pythops/gitops-demo-app
  ref:
    branch: master
  secretRef:
    name: ssh-credentials
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: gitops-demo-app
  namespace: flux-system
spec:
  interval: 30s
  sourceRef:
    kind: GitRepository
    name: gitops-demo-app
  path: ./kustomize
  prune: true
  targetNamespace: default
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: image-repo-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: gitops-demo-app
  filterTags:
    pattern: "^master-[a-f0-9]+-(?P<ts>[0-9]+)"
    extract: "$ts"
  policy:
    numerical:
      order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: gitops-demo-app
  namespace: flux-system
spec:
  image: pythops/gitops
  interval: 1m
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: gitops-demo-app
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: gitops-demo-app
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: |
        Automated image updated by Flux
        [ci skip]
    push:
      branch: master
  update:
    path: ./kustomize
    strategy: Setters

Let's see what each section is doing

GitRepository

This tells Flux from which git repository to fetch the app.

  ref:
    branch: master
  url: ssh://git@github.com/pythops/gitops-demo-app

Flux will try the fetch the master branch from the repo defined in the url

Kustomization

This indicates that we're using kustomize to deploy the app. That's why we needed the option --components=kustomize-controller

  path: ./kustomize
  targetNamespace: default

Flux will use the manifests defined in the kustomize folder and deploy them to the default namespace.

ImageRepository

This scans the image repository and fetch the tag based on the policy that we define.

  image: pythops/gitops
  interval: 1m

Flux will scan the image registry pythops/gitops each minute and fetch the image tags based on the defined ImagePolicy

ImagePolicy

  filterTags:
    pattern: "^master-[a-f0-9]+-(?P<ts>[0-9]+)"
    extract: "$ts"
  policy:
    numerical:
      order: asc

For each image tag, Flux will extract the timestamp field, and filter them by ascending order. This has the result to fetch the latest built image.

ImageUpdateAutomation

This patches the kustomize manifests with the right image tag that was fetched with the ImagePolicy

  messageTemplate: |
        Automated image updated by Flux
        [ci skip]
  ...
  update:
    path: ./kustomize

Flux will go through the files in kustomize folder, and patches the tags where it finds {"$imagepolicy": "flux-system:image-repo-policy"} with the right tag.

Notice here that we're adding [ci skip] in the commit message so we don't trigger another build that would trigger another patch commit that would trigger a build and so on ...

Once we push the file to the repository, Flux will start deploying the app.

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
gitops-demo-app-864b7548b6-f4xnn   1/1     Running   0          24s

Let's test the web app:

$ http http://gitops-demo-app
{
    "version": "v1"
}

From then, we just need to focus on building a better app and let Flux do the rest.

What's next ?

As you can see, Flux makes implementing GitOps very simple.

In the next post, I'll go through a very intresting tool called Flagger to implement progressive delivery, so stay tuned ;)

PythOps