github

Flux vs ArgoCD

I have been trying to find any comparison between FluxCD and ArgoCD that could help me choose between them. I haven’t found any that truly help us at Playtomic (many were just a list of meaningless comments if you don’t know both systems). So… we tried both and made our own decision.

Goals

We are adapting the GitOps practices. After digging into the topic, our ideal setup is:

  • The cluster (infrastructure + applications) has to be described as code, not as commands.
  • That makes the cluster easy to replicate (eases the process to set up new environments or the disaster recovery of the existing ones).
  • As few steps to build a new cluster as possible (how many actions we have to perform manually before the cluster starts running). Spoiler: you can make it with 2 commands: one to build the K8s cluster, another to install all your infra and apps.
  • The cluster must monitor changes in our repositories and automatically sync them to the cluster.
  • One repository to describe the infrastructure (cluster repo).
  • One repository to every service/application we want to deploy in the cluster (applications repo).

These two last items are called multi-tenancy. The goal is that every team can be responsible for the lifecycle of their applications: they can write anything in their repositories, but they cannot alter the cluster infrastructure by themselves. That’s the responsibility of the owners of the cluster repo.

Long story short: you can do everything in that list with both Flux and ArgoCD but there are differences about how they attain that.

Flux vs ArgoCD

Concepts

Flux:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: service-one
  namespace: playtomic
spec:
  ref:
    branch: develop 
  interval: 1m
  url: https://github.com/your-org/your-repo
  secretRef:
    name: git-credentials
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: service-one
  namespace: playtomic
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: service
  path: deploy/develop
  prune: true

ArgoCD

  • You have to install Applications. An Application is basically a repo to monitor. It supports Kustomize, Helm, …
  • An Application can contain references to more Applications.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: service-one
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/your-repo
    targetRevision: develop
    path: deploy/develop

  destination:
    server: https://kubernetes.default.svc
    namespace: playtomic

  # Sync policy
  syncPolicy:
    automated: {}

Main differences

  • ArgoCD objects have to be installed in the namespace where ArgoCD is installed (argocd in this example). That is the Application object belongs to the ArgoCD namespace. The actual application can be deployed in any target namespace.
  • Flux objects belong to the namespace where the application will be actually deployed.
  • ArgoCD supports defining several projects with different permissions (what namespaces or Kubernetes objects the application in that project are allowed to use).

How to start the cluster

Flux:

  • You write your cluster using standard k8s objects and Flux objects to indicate what pieces it has to monitor. Dependencies are written as you would write the declarative files in a regular k8s setup.
  • You have to run flux bootstrap command, and everything is set up and monitored.

ArgoCD:

  • You have to install ArgoCD (using the regular k8s descriptors).
  • You have to make ArgoCD “self-aware” after that. That is, you have to install ArgoCD as an application in ArgoCD. You will need a secret to let ArgoCD access your repositories.
  • Or use ArgoCD Autopilot (which takes care of both steps)

Deploying new versions of your services

You probably have a CI already building your images. You can let that CI commit that version to your repository too (be aware of triggered builds afterward) or you can let Flux/ArgoCD check for new images and do that commit for you.

We wanted to stop our Jenkins CI to deploy actively our services. That’s the CD responsibility and besides, the version deployed was not written anywhere: we had to check our monitor tools or query directly the cluster. So… let’s see what Flux and ArgoCD have to say about this.

Flux

You have to add two extra components when bootstrapping Flux:

flux bootstrap github --owner=your-org --repository=your-cluster-repo --token-auth --path=/clusters/develop --components-extra=image-reflector-controller,image-automation-controller

In every repository where you are storing image versions, you have to add a ImageRepository, a ImagePolicy and a ImageUpdateAutomation.

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: service-one
  namespace: playtomic
spec:
  image: your-docker-prefix/one-service
  interval: 1m0s
  secretRef:
    name: nexus-credentials
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: service-one
  namespace: playtomic
spec:
  imageRepositoryRef:
    name: configuration
  filterTags:
    pattern: '^develop-(?P<version>.*)'
    extract: '$version'
  policy:
     numerical:
      order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: service-one
  namespace: playtomic
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: one-service
  git:
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
  update:
    path: /deploy/develop/
    strategy: Setters

ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: configuration-service
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: service-one=your-docker-prefix/service-one
    argocd-image-updater.argoproj.io/service-one.allow-tags: regexp:^develop-[0-9a-zA-Z\-]+$
    argocd-image-updater.argoproj.io/service-one.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/github-credentials
    argocd-image-updater.argoproj.io/write-back-target: kustomization
    argocd-image-updater.argoproj.io/git-branch: develop
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/your-service
    targetRevision: develop
    path: deploy/develop

  ...

You have several strategies, but here we are setting write-back-target: kustomization so it will write the image versions into the kustomization.yaml in deploy/develop directory.

Main differences

  • Flux needs two extra components when bootstrapping and then an image-updater per repository whose version is monitoring.
  • ArgoCD handles everything in the Application. Flux requires a Kustomization and a GitRepository.
  • Flux’s image updater is a component in your tenant repository. ArgoCD’s updater is configured via annotation in their Application object.
  • ArgoCD has a web console (with user management). Flux does not. We found that ArgoCD web UI is super useful for not experts to debug the errors in their projects.
  • Flux bootstrap command install and make the system monitor itself. ArgoCD does not by default, you will need the ArgoCD autopilot.

Dependencies control

Sometimes you need to install some components in Kubernetes before others. In ArgoCD they managed that dependency order as waves. In Flux, by default, it retries until everything is ready (at least in our experience)

Summary

  • In both systems, it feels like the documentation was written expecting that you have already mastered the topic.
  • Flux seems more k8s-ish: everything looks like a k8s object. More raw for my taste.
  • ArgoCD seems more refined: Applications seem better defined.
  • Flux Auto Update is tricky to configure (you have to configure it in two repositories).
  • ArgoCD Auto Update seems smoother to set up (it’s a property of the Application).
  • ArgoCD has a web console while Flux does not: it makes errors debugging much simpler.

How to decide

If you have time, try both and check what adapts best to your case. I know, you don’t have the time and that’s why you are here.

For us, the decisive points are:

  • The image updater configuration is simpler in ArgoCD.
  • The image updater in Flux was killing our Nexus with requests: last versions of the updater were unable to query by image name, they have to query everything and then filter the images. As we had several services, one image updater per service, it was too much.
  • The ArgoCD console lets our team migrate their services and be able to check their deployments easily.
  • Question of taste: ArgoCD descriptors seem more friendly to me.