Proxmox to Kubernetes Migration

This documentation is partially complete, I had some issues with migration and this doesn't reflect all of it.

I've had the idea of moving away from Proxmox for a while now, and I regret moving to it in the first place. It is without a doubt good software, but it does not fit my use-case at all. The only Proxmox feature I use is KVM in extraordinary circumstances, such as when my CPU pressure is too high to initiate SSH connections. This can be solved with a hardware KVM switch and I may purchase one moving forwards.

I'm also not particularly fond of how Proxmox handles virtualisation. LXC has rarely been useful to me and working with NixOS, there's not much reason to be running more than one VM. Proxmox feels like an extra door to step through, which is why I would like to step away.

This is where I introduce Kubernetes, a container orchestration platform designed to automate the deployment, scaling, and management of containerized applications.

Kubernetes fits my use-case better as I have two nodes, and may expand in the future. I would like a structure where adding a node becomes a trivial process, where services automatically scale across a pool of resources. HA is not one of my requirements, as with k3s there is only one control plane, making it pointless.

Section 1 - Planning

Managing Downtime

I actually use my homelab for practical purposes, unlike some. I need my vaultwarden, jellyfin and a few other services to work in order to maintain my sanity. I figured then that the best way to work on this migration would be to first build the infrastructure inside of proxmox VMs and then leverage NixOS' declarative nature to swifly deploy it on bare metal.

Managing Storage

I plan to use Longhorn, developed by the team at Rancher Labs, for my primary storage. In this block storage I will keep databases, cache, and configuration. I don't have much SSD storage in my nodes so I will keep large items out of volumes. I will use a replication factor of two, ensuring I have a safety net for corruption.

For large items such as media and files, I will use either Samba or NFS. My current homelab uses Samba, but I will weigh up the benefits of both and decide at a later date.

GitOps / Manifests / Helm Charts

I will use FluxCD to apply my configuration to the cluster. Flux acts as an automated reconciler which constantly pulls changes from a Git repository containing manifests and applying them to ensure the live cluster matches the state declared in the code. I will convert all my services to raw manifests before using it, avoiding Helm charts where possible.

I made this call as I would rather know exactly what is running in my cluster instead of relying on another layer of abstraction and maintenance.

Section 2 - Migration

Backup

There are two physical nodes in my current homelab named Azalea and Clover. I had set up Kubernetes inside two NixOS VMs on Azalea to ensure I could get something working.

To start the migration I moved both of the VMs to Clover because I wanted Azalea to become my control plane, which means I needed to wipe Proxmox off of it first.

After both VMs were on Clover, I attached all the volumes I wanted to back up to pods, selected them in the Longhorn UI and created a systemRestore.

Installation

Azalea is now renamed to Fern, as this is the name of the control plane VM. I wiped Azalea and installed NixOS on it.

I start by using croc to copy my hardware-configuration.nix into my infra repository, which contains the flake for my homelab nodes.

I then set up sops-nix keys and connected the node to my tailnet with the same hostname and IP that the VM used.

Longhorn

Once these prerequisites were complete, I installed the Longhorn Helm chart with the following command:

helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --create-namespace \
  --version 1.11.1 # This should reflect the version of Longhorn that the systemRestore was created on.

I need to add a secret so I can authenticate with my CIFS share:

apiVersion: v1
kind: Secret
metadata:
  name: cifs-secret
  namespace: longhorn-system
type: Opaque
stringData:
  CIFS_USERNAME: <username>
  CIFS_PASSWORD: <password>

kubectl apply -f cifs-secret.yaml

I then forwardthe Longhorn UI, add my CIFS share and restore from the systemRestore backup.

This may take a while.

Cert Manager

Next, I have to set up cert-manager from it's Helm chart:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

I also have to add my Cloudflare token:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-token
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <token>

kubectl apply -f cloudflare-token.yaml

FluxCD

Afterwards, I set up FluxCD with the following commands:

export GITHUB_TOKEN=<token>
flux bootstrap github \
  --owner=qazer2687 \
  --repository=manifests \
  --branch=main \
  --path=clusters/home

Longhorn & Flux Issues

This section was written after the fact and is quite long as I'm using the actual commands I used to fix this so that I can refer to this in the future if I need to.

After Longhorn performed its systemRestore, the PVs were recreated but the PVCs were not. This led to Flux failing to reconcile because it attempts to apply PVC manifests with volumeName: "" against bound PVCs, which Kubernetes rejects as an immutable field change.

The fix was to suspend Flux, manually recreate the PVCs with the correct volumeName, annotate them to prevent Flux from modifying the spec, then resume Flux.

I start by suspending Flux:

flux suspend kustomization --all

I then patch the claimRef on each PV:

kubectl patch pv pvc-51f71165-04ca-4a9c-96c4-84413b2ad02f --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"forgejo-data","namespace":"apps"}}}'
kubectl patch pv pvc-ee0849ce-2262-4acb-b825-4a57d3e15ccc --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"vaultwarden-data","namespace":"apps"}}}'
kubectl patch pv pvc-6fe23d50-720b-42f5-9502-57dffd57960b --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"miniflux-postgres-data","namespace":"apps"}}}'
kubectl patch pv pvc-c6fff0c3-6f03-403d-8f90-d113fc4e01bf --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"openwebui-data","namespace":"apps"}}}'
kubectl patch pv pvc-5bfd32f8-5a35-4718-ae77-44720be81df4 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"beszel-data","namespace":"infra"}}}'
kubectl patch pv pvc-25d18db3-548f-433d-8c8b-cd3062d75320 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"bazarr-data","namespace":"media"}}}'
kubectl patch pv pvc-c1f12161-2b51-4702-83bf-79a66dedc6c9 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"immich-postgres-data","namespace":"media"}}}'
kubectl patch pv pvc-4b64a207-8964-48f8-baad-a5f14597e46c --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"immich-redis-data","namespace":"media"}}}'
kubectl patch pv pvc-2eb2aa9e-a3b3-405d-b191-c70a30b9bfb9 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"jellyfin-data","namespace":"media"}}}'
kubectl patch pv pvc-f1c7c731-b4b4-40c0-9e68-abcf41672fee --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"prowlarr-data","namespace":"media"}}}'
kubectl patch pv pvc-f60e46e3-7b48-4761-81a1-ff4d34f501c8 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"qbittorrent-data","namespace":"media"}}}'
kubectl patch pv pvc-1c14b613-d42f-4e31-9339-894eee2cb650 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"qui-data","namespace":"media"}}}'
kubectl patch pv pvc-5059ff1e-5827-4fd6-a379-f71d2048f530 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"radarr-data","namespace":"media"}}}'
kubectl patch pv pvc-91f75cea-cd2b-43f7-ae69-775b3918b6cb --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"sonarr-data","namespace":"media"}}}'
kubectl patch pv pvc-112d8b73-345c-4a01-8d09-b86efe564841 --type=merge -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"warp-data","namespace":"media"}}}'

Then create the PVCs:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: forgejo-data
  namespace: apps
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-51f71165-04ca-4a9c-96c4-84413b2ad02f
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vaultwarden-data
  namespace: apps
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-ee0849ce-2262-4acb-b825-4a57d3e15ccc
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: miniflux-postgres-data
  namespace: apps
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-6fe23d50-720b-42f5-9502-57dffd57960b
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openwebui-data
  namespace: apps
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-c6fff0c3-6f03-403d-8f90-d113fc4e01bf
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: beszel-data
  namespace: infra
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-5bfd32f8-5a35-4718-ae77-44720be81df4
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bazarr-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-25d18db3-548f-433d-8c8b-cd3062d75320
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: immich-postgres-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-c1f12161-2b51-4702-83bf-79a66dedc6c9
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: immich-redis-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-4b64a207-8964-48f8-baad-a5f14597e46c
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jellyfin-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-2eb2aa9e-a3b3-405d-b191-c70a30b9bfb9
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prowlarr-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-f1c7c731-b4b4-40c0-9e68-abcf41672fee
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qbittorrent-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-f60e46e3-7b48-4761-81a1-ff4d34f501c8
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qui-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-1c14b613-d42f-4e31-9339-894eee2cb650
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: radarr-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-5059ff1e-5827-4fd6-a379-f71d2048f530
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonarr-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-91f75cea-cd2b-43f7-ae69-775b3918b6cb
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: warp-data
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-retain
  volumeName: pvc-112d8b73-345c-4a01-8d09-b86efe564841
  resources:
    requests:
      storage: 1Gi
EOF

Then I verify all PVCs are bound:

kubectl get pvc -A -o wide

I annotate all the PVCs to prevent Flux reconciling the spec:

kubectl annotate pvc forgejo-data -n apps kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc vaultwarden-data -n apps kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc miniflux-postgres-data -n apps kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc openwebui-data -n apps kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc beszel-data -n infra kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc bazarr-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc immich-postgres-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc immich-redis-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc jellyfin-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc prowlarr-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc qbittorrent-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc qui-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc radarr-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc sonarr-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite
kubectl annotate pvc warp-data -n media kustomize.toolkit.fluxcd.io/reconcile=disabled --overwrite

Finally I resume Flux and verify everything works:

flux resume kustomization --all
flux get kustomizations
kubectl get pods -A

To document new PV UUIDs, I can use the command below, which puts them in a table.

kubectl get pv -o custom-columns='NAME:.metadata.name,CLAIM:.spec.claimRef.name,NS:.spec.claimRef.namespace,SIZE:.spec.capacity.storage'

Section 3 - Afterthoughts

Kubernetes is hard. Before I switched I wonderered why there was so much fearmongering surrounding k8s. I figured that because I was using k3s and using a small subset of k8s features that I would be better off, but I was mistaken.

There are still a lot of features/terms that I needed to have an understanding of to get anywhere such as: PVs, PVCs, Pods, Containers, Ingress, Services, Deployments, CSIs, Helm (for Flux and cert-manager), StatefulSets, DaemonSets, Namespaces, Secrets and probably a lot more I'm missing, not to mention manifest syntax.

Kubernetes absolutely has a steep learning curve, it's not like Docker where you just copy-paste the provided compose file, change a few variables and run it. I'm sure it could be relatively close in some aspects, like with Helm, but I'm not using Helm. I needed to manually convert each compose file into a manifest, I did this without Kompose as I found it not to be very helpful. It didn't take too long to learn the fundementals of this though, so it was worth the initial friction.

What I'm still struggling to justify the worth of is Longhorn. Longhorn has been very painful for me, I think partially because I only have two nodes. Longhorn seems to absolutely hate resource pressure or a node going offline. I've had significant trouble with the backup process too. The systemRestore I took seemed to discard any PV to PVC bindings and I had to manually attach them all back, which broke FluxCD, though I covered the fix earlier in this documentation. Perhaps I did something wrong, but my confidence in Longhorn is not high at the moment.

NFS on a NAS would relieve a significant amount of pain, but HDD prices make it hard to justify building one. I don't use very much storage either, I'm currently sitting at around a terrabyte, I don't have any desire to hold onto entertainment media either which is where about half of that storage is currently assigned.

The biggest difference in my workflow is commitment. If I wanted to set something up on Docker, I could simply grap the provided compose file (which there almost always was), change a couple things that I was familiar with and then spin it up and test things out. I ran Docker on a single node and preferred to bind mount everything to a single directory, which was easy to back up.

On Kubernetes, setting up a service feels like a commitment, I have to really know that I want to set this up otherwise it's not going to be worth the time.

With all this being said, I don't hate Kubernetes. I'm going to keep it for a while and I may revisit this documentation if I develop the cluster any more.