# imcu.ro HA Kubernetes Cluster 3-node highly available Kubernetes cluster running on Hetzner Cloud, connected via Netbird VPN mesh, with distributed storage via Longhorn. ## Cluster Nodes | Node | Public IP | VPN IP (wt0) | Location | Volume | |---|---|---|---|---| | k3s-fi-01 | 89.167.46.12 | 100.121.136.204 | Helsinki, FI | scsi-0HC_Volume_106631386 | | k3s-de-01 | 91.99.15.157 | 100.121.191.170 | Falkenstein, DE | scsi-0HC_Volume_106631392 | | k3s-de-02 | 2.28.55.101 | 100.121.49.137 | Falkenstein, DE | scsi-0HC_Volume_106631403 | All nodes: Ubuntu 26.04 LTS, 2 vCPU, 3.7 Gi RAM, 100 GB Hetzner volume (mounted at `/var/lib/longhorn`). ## Network Topology ``` Internet | Cloudflare DNS (imcu.ro zone) | +---------------+---------------+ | | | 89.167.46.12 91.99.15.157 2.28.55.101 k3s-fi-01 k3s-de-01 k3s-de-02 | | | +------- wt0 (Netbird VPN) -----+ 100.121.x.x mesh | k3s control plane (embedded etcd, HA) ``` - **All k3s traffic** (API server, etcd, flannel, node communication) runs over the Netbird VPN (`wt0` interface). - **Public IPs** are only used for: ingress traffic (Traefik LoadBalancer), Gitea SSH (NodePort 30022), and SSH admin access. - **Netbird control plane** runs on `167.233.140.254` at `netbird.imcu.ro`. ## Core Components | Component | Version | Purpose | |---|---|---| | k3s | v1.36.3+k3s1 | Kubernetes distribution with embedded etcd (HA) | | Longhorn | v1.12.1 | Distributed block storage (2 replicas, `/var/lib/longhorn/`) | | Traefik | 3.x | Ingress controller (3 replicas, LoadBalancer on all node IPs) | | cert-manager | v1.18.2 | TLS certificate management (Let's Encrypt, auto-renewing) | All three are installed as k3s-native `HelmChart` resources in `kube-system` namespace. ## Storage - **Longhorn** is the default StorageClass (`longhorn`). - Each node contributes its 100 GB Hetzner volume as a Longhorn disk at `/var/lib/longhorn/`. - Volumes are replicated across **2 of 3 nodes** (configurable per-PVC). - To add more disks: mount a new volume at `/mnt/longhorn-disk2`, add it via Longhorn UI. ### Persistent Volumes | PVC | Namespace | Size | Content | |---|---|---|---| | gitea-data | gitea | 10 Gi | Gitea SQLite DB, git repos, packages | | wolsey-content | default | 10 Gi | Wolsey static content (6.5 GB) | | timescaledb-data | system | 10 Gi | TimescaleDB data | ## Workloads | Service | URL | Namespace | Image | Storage | |---|---|---|---|---| | Gitea | https://gitea.imcu.ro | gitea | gitea/gitea:1.27.2-rootless | 10 GB Longhorn | | Gitea SSH | gitea.imcu.ro:30022 | gitea | (same pod) | NodePort 30022 | | Gitea Registry | gitea.imcu.ro/v2 | gitea | (built-in) | — | | Wolsey | https://wolsey.imcu.ro | default | homeschool/school:e76d244 | 10 GB Longhorn | | School Games (prod) | https://school.imcu.ro | games | homeschool/school-games:20260112T132336 | — | | School Games (dev) | https://develop.imcu.ro | default | homeschool/games:7d3a57d | — | | School Games (default) | — | default | homeschool/games:2a4f03df | — | | Randomly | https://randomly.imcu.ro | randomly | homeschool/randomly:latest | — | | TimescaleDB | timescaledb.system.svc:5432 | system | timescale/timescaledb:2.29.1-pg16 | 10 GB Longhorn | ## DNS (Cloudflare) All A records point to all 3 node public IPs (round-robin): | Record | Type | Value | |---|---|---| | gitea.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 | | school.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 | | wolsey.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 | | develop.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 | | randomly.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 | | *.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 | | netbird.imcu.ro | A | 167.233.140.254 | ## TLS Certificates All managed by cert-manager via `letsencrypt-prod` ClusterIssuer (HTTP-01 challenge via Traefik). Auto-renewed 30 days before expiry. | Domain | Secret | Namespace | |---|---|---| | gitea.imcu.ro | gitea-tls | gitea | | wolsey.imcu.ro | wolsey-imcu-ro-tls | default | | school.imcu.ro | school-imcu-ro-tls | games | | develop.imcu.ro | develop-imcu-ro-tls | default | | randomly.imcu.ro | randomly-imcu-ro-tls | randomly | ## Admin Access ### SSH All k3s nodes have SSH keys distributed. From any node you can SSH to any other using VPN IPs: ``` ssh k3s-fi-01 # 100.121.136.204 ssh k3s-de-01 # 100.121.191.170 ssh k3s-de-02 # 100.121.49.137 ssh school # 157.90.228.70 (old server, public IP) ``` ### kubectl ```bash ssh root@89.167.46.12 export KUBECONFIG=/etc/rancher/k3s/k3s.yaml k3s kubectl get nodes ``` ### Longhorn UI SSH tunnel (UI is not publicly exposed): ```bash ssh -L 8080:10.43.255.96:80 root@89.167.46.12 # Open http://localhost:8080 ``` ### Gitea - Web: https://gitea.imcu.ro - SSH: `git clone ssh://git@gitea.imcu.ro:30022/homeschool/repo.git` - Registry: `docker login gitea.imcu.ro` (use Gitea access token as password) ## Old Server (school) `157.90.228.70` (hostname `school`) is the previous single-node k3s cluster. All workloads have been migrated to the HA cluster. It still runs only system pods (kube-system, cert-manager) and preserves original data as backup: | Path | Size | Content | |---|---|---| | /opt/gitea/ | 255 MB | Gitea data backup | | /opt/registry-data/ | 411 MB | Old standalone registry | | /mnt/HC_Volume_106055137/school-content/ | 6.5 GB | Wolsey content backup | Can be decommissioned or repurposed once the migration is confirmed stable. ## Bootstrap (reference) The cluster was built in this order: 1. Netbird client installed on all 3 nodes, enrolled with setup key, tunnels up on `wt0` 2. Hetzner volumes mounted at `/var/lib/longhorn` on all 3 nodes (fstab updated) 3. k3s installed with `--cluster-init` on k3s-fi-01, then joined k3s-de-01 and k3s-de-02 4. Longhorn installed via k3s HelmChart (2 replicas, default data path `/var/lib/longhorn/`) 5. Traefik installed via k3s HelmChart (3 replicas, LoadBalancer) 6. cert-manager installed via k3s HelmChart, ClusterIssuer created 7. Gitea deployed, data migrated from school via tar over SSH 8. All other workloads migrated from school, images pushed to Gitea registry ### k3s install flags **Node 1 (cluster-init):** ```bash curl -sfL https://get.k3s.io | sh -s - server \ --cluster-init \ --node-ip 100.121.136.204 \ --node-external-ip 89.167.46.12 \ --flannel-iface wt0 \ --advertise-address 100.121.136.204 \ --tls-san netbird.imcu.ro \ --tls-san 100.121.136.204 \ --tls-san 100.121.191.170 \ --tls-san 100.121.49.137 \ --disable traefik ``` **Nodes 2 & 3 (join):** ```bash curl -sfL https://get.k3s.io | sh -s - server \ --server https://100.121.136.204:6443 \ --token \ --node-ip \ --node-external-ip \ --flannel-iface wt0 \ --disable traefik ``` ## SSH Key Mesh All k3s nodes have ed25519 SSH keys distributed to each other and to school. Config at `/root/.ssh/config` on each node uses VPN IPs for k3s nodes and public IP for school. ``` Host k3s-fi-01 HostName 100.121.136.204 Host k3s-de-01 HostName 100.121.191.170 Host k3s-de-02 HostName 100.121.49.137 Host school HostName 157.90.228.70 ``` ## Repository Structure (GitOps) This repo lives at `gitea.imcu.ro/cluster/k3s-cluster`. It contains all manifests needed to recreate the cluster from scratch. ``` k3s-cluster/ ├── README.md # This file (architecture overview) ├── AGENTS.md # Operational reference (per-service details) ├── deploy.sh # Apply all manifests to the cluster ├── .gitignore ├── helmcharts/ # k3s HelmChart resources (system components) │ ├── traefik.yaml # Ingress controller (3 replicas, LoadBalancer) │ ├── cert-manager.yaml # TLS certificate manager (Let's Encrypt) │ └── longhorn.yaml # Distributed storage (2 replicas, /var/lib/longhorn/) ├── cluster-config/ # Cluster-wide configuration │ ├── namespaces.yaml # Application namespaces (gitea, games, randomly, system) │ ├── clusterissuer.yaml # Let's Encrypt ClusterIssuer │ └── image-pull-secret-template.txt # How to create gitea-registry secrets └── workloads/ # Application manifests (deploy + svc + ingress + pvc) ├── gitea/gitea.yaml ├── wolsey/wolsey.yaml ├── school-games-prod/school-games-prod.yaml ├── school-games-dev/school-games-dev.yaml ├── school-games-default/school-games-default.yaml ├── randomly/randomly.yaml └── timescaledb/timescaledb.yaml ``` ### How to update a service 1. Edit the manifest in `workloads//.yaml` 2. Apply: `k3s kubectl apply -f workloads//.yaml` 3. Commit and push to keep the repo in sync ### How to update a HelmChart (system component) 1. Edit the manifest in `helmcharts/.yaml` (e.g. bump `version:` for cert-manager) 2. Apply: `k3s kubectl apply -f helmcharts/.yaml` 3. k3s HelmChart controller will automatically reconcile the change 4. Commit and push ### How to deploy from scratch ```bash # From any k3s node: KUBECONFIG=/etc/rancher/k3s/k3s.yaml ./deploy.sh ```