imcu.ro HA Kubernetes Cluster

3-node highly available Kubernetes cluster running on Hetzner Cloud, connected via Netbird VPN mesh, with distributed storage via Longhorn.

Cluster Nodes

Node	Public IP	VPN IP (wt0)	Location	Volume
k3s-fi-01	89.167.46.12	100.121.136.204	Helsinki, FI	scsi-0HC_Volume_106631386
k3s-de-01	91.99.15.157	100.121.191.170	Falkenstein, DE	scsi-0HC_Volume_106631392
k3s-de-02	2.28.55.101	100.121.49.137	Falkenstein, DE	scsi-0HC_Volume_106631403

All nodes: Ubuntu 26.04 LTS, 2 vCPU, 3.7 Gi RAM, 100 GB Hetzner volume (mounted at /var/lib/longhorn).

Network Topology

                        Internet
                           |
                    Cloudflare DNS
                   (imcu.ro zone)
                           |
           +---------------+---------------+
           |               |               |
      89.167.46.12   91.99.15.157   2.28.55.101
        k3s-fi-01       k3s-de-01       k3s-de-02
           |               |               |
           +------- wt0 (Netbird VPN) -----+
                   100.121.x.x mesh
                           |
                    k3s control plane
                    (embedded etcd, HA)

• All k3s traffic (API server, etcd, flannel, node communication) runs over the Netbird VPN (wt0 interface).
• Public IPs are only used for: ingress traffic (Traefik LoadBalancer), Gitea SSH (NodePort 30022), and SSH admin access.
• Netbird control plane runs on 167.233.140.254 at netbird.imcu.ro.

Core Components

Component	Version	Purpose
k3s	v1.36.3+k3s1	Kubernetes distribution with embedded etcd (HA)
Longhorn	v1.12.1	Distributed block storage (2 replicas, /var/lib/longhorn/)
Traefik	3.x	Ingress controller (3 replicas, LoadBalancer on all node IPs)
cert-manager	v1.18.2	TLS certificate management (Let's Encrypt, auto-renewing)

All three are installed as k3s-native HelmChart resources in kube-system namespace.

Storage

• Longhorn is the default StorageClass (longhorn).
• Each node contributes its 100 GB Hetzner volume as a Longhorn disk at /var/lib/longhorn/.
• Volumes are replicated across 2 of 3 nodes (configurable per-PVC).
• To add more disks: mount a new volume at /mnt/longhorn-disk2, add it via Longhorn UI.

Persistent Volumes

PVC	Namespace	Size	Content
gitea-data	gitea	10 Gi	Gitea SQLite DB, git repos, packages
wolsey-content	default	10 Gi	Wolsey static content (6.5 GB)

Workloads

Service	URL	Namespace	Image	Storage
Gitea	https://gitea.imcu.ro	gitea	gitea/gitea:1.27.2-rootless	10 GB Longhorn
Gitea SSH	gitea.imcu.ro:30022	gitea	(same pod)	NodePort 30022
Gitea Registry	gitea.imcu.ro/v2	gitea	(built-in)	—
Wolsey	https://wolsey.imcu.ro	default	homeschool/school:e76d244	10 GB Longhorn
School Games (prod)	https://school.imcu.ro	games	homeschool/school-games:20260112T132336	—
School Games (dev)	https://develop.imcu.ro	default	homeschool/games:7d3a57d	—
School Games (default)	—	default	homeschool/games:2a4f03df	—
Randomly	https://randomly.imcu.ro	randomly	homeschool/randomly:latest	—

DNS (Cloudflare)

All A records point to all 3 node public IPs (round-robin):

Record	Type	Value
gitea.imcu.ro	A x3	89.167.46.12, 91.99.15.157, 2.28.55.101
school.imcu.ro	A x3	89.167.46.12, 91.99.15.157, 2.28.55.101
wolsey.imcu.ro	A x3	89.167.46.12, 91.99.15.157, 2.28.55.101
develop.imcu.ro	A x3	89.167.46.12, 91.99.15.157, 2.28.55.101
randomly.imcu.ro	A x3	89.167.46.12, 91.99.15.157, 2.28.55.101
*.imcu.ro	A x3	89.167.46.12, 91.99.15.157, 2.28.55.101
netbird.imcu.ro	A	167.233.140.254

TLS Certificates

All managed by cert-manager via letsencrypt-prod ClusterIssuer (HTTP-01 challenge via Traefik). Auto-renewed 30 days before expiry.

Domain	Secret	Namespace
gitea.imcu.ro	gitea-tls	gitea
wolsey.imcu.ro	wolsey-imcu-ro-tls	default
school.imcu.ro	school-imcu-ro-tls	games
develop.imcu.ro	develop-imcu-ro-tls	default
randomly.imcu.ro	randomly-imcu-ro-tls	randomly

Admin Access

SSH

All k3s nodes have SSH keys distributed. From any node you can SSH to any other using VPN IPs:

ssh k3s-fi-01   # 100.121.136.204
ssh k3s-de-01   # 100.121.191.170
ssh k3s-de-02   # 100.121.49.137
ssh school      # 157.90.228.70 (old server, public IP)

kubectl

ssh root@89.167.46.12
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
k3s kubectl get nodes

Longhorn UI

SSH tunnel (UI is not publicly exposed):

ssh -L 8080:10.43.255.96:80 root@89.167.46.12
# Open http://localhost:8080

Gitea

• Web: https://gitea.imcu.ro
• SSH: git clone ssh://git@gitea.imcu.ro:30022/homeschool/repo.git
• Registry: docker login gitea.imcu.ro (use Gitea access token as password)

Old Server (school)

157.90.228.70 (hostname school) is the previous single-node k3s cluster. All workloads have been migrated to the HA cluster. It still runs only system pods (kube-system, cert-manager) and preserves original data as backup:

Path	Size	Content
/opt/gitea/	255 MB	Gitea data backup
/opt/registry-data/	411 MB	Old standalone registry
/mnt/HC_Volume_106055137/school-content/	6.5 GB	Wolsey content backup

Can be decommissioned or repurposed once the migration is confirmed stable.

Bootstrap (reference)

The cluster was built in this order:

1. Netbird client installed on all 3 nodes, enrolled with setup key, tunnels up on wt0
2. Hetzner volumes mounted at /var/lib/longhorn on all 3 nodes (fstab updated)
3. k3s installed with --cluster-init on k3s-fi-01, then joined k3s-de-01 and k3s-de-02
4. Longhorn installed via k3s HelmChart (2 replicas, default data path /var/lib/longhorn/)
5. Traefik installed via k3s HelmChart (3 replicas, LoadBalancer)
6. cert-manager installed via k3s HelmChart, ClusterIssuer created
7. Gitea deployed, data migrated from school via tar over SSH
8. All other workloads migrated from school, images pushed to Gitea registry

k3s install flags

Node 1 (cluster-init):

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --node-ip 100.121.136.204 \
  --node-external-ip 89.167.46.12 \
  --flannel-iface wt0 \
  --advertise-address 100.121.136.204 \
  --tls-san netbird.imcu.ro \
  --tls-san 100.121.136.204 \
  --tls-san 100.121.191.170 \
  --tls-san 100.121.49.137 \
  --disable traefik

Nodes 2 & 3 (join):

curl -sfL https://get.k3s.io | sh -s - server \
  --server https://100.121.136.204:6443 \
  --token <node-token> \
  --node-ip <vpn-ip> \
  --node-external-ip <public-ip> \
  --flannel-iface wt0 \
  --disable traefik

SSH Key Mesh

All k3s nodes have ed25519 SSH keys distributed to each other and to school. Config at /root/.ssh/config on each node uses VPN IPs for k3s nodes and public IP for school.

Host k3s-fi-01    HostName 100.121.136.204
Host k3s-de-01    HostName 100.121.191.170
Host k3s-de-02    HostName 100.121.49.137
Host school       HostName 157.90.228.70

Repository Structure (GitOps)

This repo lives at gitea.imcu.ro/cluster/k3s-cluster. It contains all manifests needed to recreate the cluster from scratch.

k3s-cluster/
├── README.md                    # This file (architecture overview)
├── AGENTS.md                    # Operational reference (per-service details)
├── deploy.sh                    # Apply all manifests to the cluster
├── .gitignore
├── helmcharts/                  # k3s HelmChart resources (system components)
│   ├── traefik.yaml             # Ingress controller (3 replicas, LoadBalancer)
│   ├── cert-manager.yaml        # TLS certificate manager (Let's Encrypt)
│   └── longhorn.yaml            # Distributed storage (2 replicas, /var/lib/longhorn/)
├── cluster-config/              # Cluster-wide configuration
│   ├── namespaces.yaml          # Application namespaces (gitea, games, randomly)
│   ├── clusterissuer.yaml       # Let's Encrypt ClusterIssuer
│   └── image-pull-secret-template.txt  # How to create gitea-registry secrets
└── workloads/                   # Application manifests (deploy + svc + ingress + pvc)
    ├── gitea/gitea.yaml
    ├── wolsey/wolsey.yaml
    ├── school-games-prod/school-games-prod.yaml
    ├── school-games-dev/school-games-dev.yaml
    ├── school-games-default/school-games-default.yaml
    └── randomly/randomly.yaml

How to update a service

1. Edit the manifest in workloads/<service>/<service>.yaml
2. Apply: k3s kubectl apply -f workloads/<service>/<service>.yaml
3. Commit and push to keep the repo in sync

How to update a HelmChart (system component)

1. Edit the manifest in helmcharts/<component>.yaml (e.g. bump version: for cert-manager)
2. Apply: k3s kubectl apply -f helmcharts/<component>.yaml
3. k3s HelmChart controller will automatically reconcile the change
4. Commit and push

How to deploy from scratch

# From any k3s node:
KUBECONFIG=/etc/rancher/k3s/k3s.yaml ./deploy.sh