mcimpoesu
fd7c529255
- 3-node HA cluster (k3s v1.36.3+k3s1, embedded etcd) on Hetzner - Netbird VPN mesh (wt0) for all cluster traffic - Longhorn distributed storage (2 replicas, /var/lib/longhorn/) - Traefik ingress (3 replicas, LoadBalancer on all node IPs) - cert-manager with Let's Encrypt (auto-renewing TLS) - Gitea (git + container registry) with SQLite on Longhorn - 4 migrated workloads: wolsey, school-games (3 variants), randomly - All HelmCharts and workload manifests as YAML - deploy.sh for one-shot cluster setup
249 lines
9.3 KiB
Markdown
249 lines
9.3 KiB
Markdown
# imcu.ro HA Kubernetes Cluster
|
|
|
|
3-node highly available Kubernetes cluster running on Hetzner Cloud, connected via Netbird VPN mesh, with distributed storage via Longhorn.
|
|
|
|
## Cluster Nodes
|
|
|
|
| Node | Public IP | VPN IP (wt0) | Location | Volume |
|
|
|---|---|---|---|---|
|
|
| k3s-fi-01 | 89.167.46.12 | 100.121.136.204 | Helsinki, FI | scsi-0HC_Volume_106631386 |
|
|
| k3s-de-01 | 91.99.15.157 | 100.121.191.170 | Falkenstein, DE | scsi-0HC_Volume_106631392 |
|
|
| k3s-de-02 | 2.28.55.101 | 100.121.49.137 | Falkenstein, DE | scsi-0HC_Volume_106631403 |
|
|
|
|
All nodes: Ubuntu 26.04 LTS, 2 vCPU, 3.7 Gi RAM, 100 GB Hetzner volume (mounted at `/var/lib/longhorn`).
|
|
|
|
## Network Topology
|
|
|
|
```
|
|
Internet
|
|
|
|
|
Cloudflare DNS
|
|
(imcu.ro zone)
|
|
|
|
|
+---------------+---------------+
|
|
| | |
|
|
89.167.46.12 91.99.15.157 2.28.55.101
|
|
k3s-fi-01 k3s-de-01 k3s-de-02
|
|
| | |
|
|
+------- wt0 (Netbird VPN) -----+
|
|
100.121.x.x mesh
|
|
|
|
|
k3s control plane
|
|
(embedded etcd, HA)
|
|
```
|
|
|
|
- **All k3s traffic** (API server, etcd, flannel, node communication) runs over the Netbird VPN (`wt0` interface).
|
|
- **Public IPs** are only used for: ingress traffic (Traefik LoadBalancer), Gitea SSH (NodePort 30022), and SSH admin access.
|
|
- **Netbird control plane** runs on `167.233.140.254` at `netbird.imcu.ro`.
|
|
|
|
## Core Components
|
|
|
|
| Component | Version | Purpose |
|
|
|---|---|---|
|
|
| k3s | v1.36.3+k3s1 | Kubernetes distribution with embedded etcd (HA) |
|
|
| Longhorn | v1.12.1 | Distributed block storage (2 replicas, `/var/lib/longhorn/`) |
|
|
| Traefik | 3.x | Ingress controller (3 replicas, LoadBalancer on all node IPs) |
|
|
| cert-manager | v1.18.2 | TLS certificate management (Let's Encrypt, auto-renewing) |
|
|
|
|
All three are installed as k3s-native `HelmChart` resources in `kube-system` namespace.
|
|
|
|
## Storage
|
|
|
|
- **Longhorn** is the default StorageClass (`longhorn`).
|
|
- Each node contributes its 100 GB Hetzner volume as a Longhorn disk at `/var/lib/longhorn/`.
|
|
- Volumes are replicated across **2 of 3 nodes** (configurable per-PVC).
|
|
- To add more disks: mount a new volume at `/mnt/longhorn-disk2`, add it via Longhorn UI.
|
|
|
|
### Persistent Volumes
|
|
|
|
| PVC | Namespace | Size | Content |
|
|
|---|---|---|---|
|
|
| gitea-data | gitea | 10 Gi | Gitea SQLite DB, git repos, packages |
|
|
| wolsey-content | default | 10 Gi | Wolsey static content (6.5 GB) |
|
|
|
|
## Workloads
|
|
|
|
| Service | URL | Namespace | Image | Storage |
|
|
|---|---|---|---|---|
|
|
| Gitea | https://gitea.imcu.ro | gitea | gitea/gitea:1.27.2-rootless | 10 GB Longhorn |
|
|
| Gitea SSH | gitea.imcu.ro:30022 | gitea | (same pod) | NodePort 30022 |
|
|
| Gitea Registry | gitea.imcu.ro/v2 | gitea | (built-in) | — |
|
|
| Wolsey | https://wolsey.imcu.ro | default | homeschool/school:e76d244 | 10 GB Longhorn |
|
|
| School Games (prod) | https://school.imcu.ro | games | homeschool/school-games:20260112T132336 | — |
|
|
| School Games (dev) | https://develop.imcu.ro | default | homeschool/games:7d3a57d | — |
|
|
| School Games (default) | — | default | homeschool/games:2a4f03df | — |
|
|
| Randomly | https://randomly.imcu.ro | randomly | homeschool/randomly:latest | — |
|
|
|
|
## DNS (Cloudflare)
|
|
|
|
All A records point to all 3 node public IPs (round-robin):
|
|
|
|
| Record | Type | Value |
|
|
|---|---|---|
|
|
| gitea.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 |
|
|
| school.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 |
|
|
| wolsey.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 |
|
|
| develop.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 |
|
|
| randomly.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 |
|
|
| *.imcu.ro | A x3 | 89.167.46.12, 91.99.15.157, 2.28.55.101 |
|
|
| netbird.imcu.ro | A | 167.233.140.254 |
|
|
|
|
## TLS Certificates
|
|
|
|
All managed by cert-manager via `letsencrypt-prod` ClusterIssuer (HTTP-01 challenge via Traefik). Auto-renewed 30 days before expiry.
|
|
|
|
| Domain | Secret | Namespace |
|
|
|---|---|---|
|
|
| gitea.imcu.ro | gitea-tls | gitea |
|
|
| wolsey.imcu.ro | wolsey-imcu-ro-tls | default |
|
|
| school.imcu.ro | school-imcu-ro-tls | games |
|
|
| develop.imcu.ro | develop-imcu-ro-tls | default |
|
|
| randomly.imcu.ro | randomly-imcu-ro-tls | randomly |
|
|
|
|
## Admin Access
|
|
|
|
### SSH
|
|
|
|
All k3s nodes have SSH keys distributed. From any node you can SSH to any other using VPN IPs:
|
|
|
|
```
|
|
ssh k3s-fi-01 # 100.121.136.204
|
|
ssh k3s-de-01 # 100.121.191.170
|
|
ssh k3s-de-02 # 100.121.49.137
|
|
ssh school # 157.90.228.70 (old server, public IP)
|
|
```
|
|
|
|
### kubectl
|
|
|
|
```bash
|
|
ssh root@89.167.46.12
|
|
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
|
|
k3s kubectl get nodes
|
|
```
|
|
|
|
### Longhorn UI
|
|
|
|
SSH tunnel (UI is not publicly exposed):
|
|
|
|
```bash
|
|
ssh -L 8080:10.43.255.96:80 root@89.167.46.12
|
|
# Open http://localhost:8080
|
|
```
|
|
|
|
### Gitea
|
|
|
|
- Web: https://gitea.imcu.ro
|
|
- SSH: `git clone ssh://git@gitea.imcu.ro:30022/homeschool/repo.git`
|
|
- Registry: `docker login gitea.imcu.ro` (use Gitea access token as password)
|
|
|
|
## Old Server (school)
|
|
|
|
`157.90.228.70` (hostname `school`) is the previous single-node k3s cluster. All workloads have been migrated to the HA cluster. It still runs only system pods (kube-system, cert-manager) and preserves original data as backup:
|
|
|
|
| Path | Size | Content |
|
|
|---|---|---|
|
|
| /opt/gitea/ | 255 MB | Gitea data backup |
|
|
| /opt/registry-data/ | 411 MB | Old standalone registry |
|
|
| /mnt/HC_Volume_106055137/school-content/ | 6.5 GB | Wolsey content backup |
|
|
|
|
Can be decommissioned or repurposed once the migration is confirmed stable.
|
|
|
|
## Bootstrap (reference)
|
|
|
|
The cluster was built in this order:
|
|
|
|
1. Netbird client installed on all 3 nodes, enrolled with setup key, tunnels up on `wt0`
|
|
2. Hetzner volumes mounted at `/var/lib/longhorn` on all 3 nodes (fstab updated)
|
|
3. k3s installed with `--cluster-init` on k3s-fi-01, then joined k3s-de-01 and k3s-de-02
|
|
4. Longhorn installed via k3s HelmChart (2 replicas, default data path `/var/lib/longhorn/`)
|
|
5. Traefik installed via k3s HelmChart (3 replicas, LoadBalancer)
|
|
6. cert-manager installed via k3s HelmChart, ClusterIssuer created
|
|
7. Gitea deployed, data migrated from school via tar over SSH
|
|
8. All other workloads migrated from school, images pushed to Gitea registry
|
|
|
|
### k3s install flags
|
|
|
|
**Node 1 (cluster-init):**
|
|
```bash
|
|
curl -sfL https://get.k3s.io | sh -s - server \
|
|
--cluster-init \
|
|
--node-ip 100.121.136.204 \
|
|
--node-external-ip 89.167.46.12 \
|
|
--flannel-iface wt0 \
|
|
--advertise-address 100.121.136.204 \
|
|
--tls-san netbird.imcu.ro \
|
|
--tls-san 100.121.136.204 \
|
|
--tls-san 100.121.191.170 \
|
|
--tls-san 100.121.49.137 \
|
|
--disable traefik
|
|
```
|
|
|
|
**Nodes 2 & 3 (join):**
|
|
```bash
|
|
curl -sfL https://get.k3s.io | sh -s - server \
|
|
--server https://100.121.136.204:6443 \
|
|
--token <node-token> \
|
|
--node-ip <vpn-ip> \
|
|
--node-external-ip <public-ip> \
|
|
--flannel-iface wt0 \
|
|
--disable traefik
|
|
```
|
|
|
|
## SSH Key Mesh
|
|
|
|
All k3s nodes have ed25519 SSH keys distributed to each other and to school. Config at `/root/.ssh/config` on each node uses VPN IPs for k3s nodes and public IP for school.
|
|
|
|
```
|
|
Host k3s-fi-01 HostName 100.121.136.204
|
|
Host k3s-de-01 HostName 100.121.191.170
|
|
Host k3s-de-02 HostName 100.121.49.137
|
|
Host school HostName 157.90.228.70
|
|
```
|
|
|
|
## Repository Structure (GitOps)
|
|
|
|
This repo lives at `gitea.imcu.ro/cluster/k3s-cluster`. It contains all manifests needed to recreate the cluster from scratch.
|
|
|
|
```
|
|
k3s-cluster/
|
|
├── README.md # This file (architecture overview)
|
|
├── AGENTS.md # Operational reference (per-service details)
|
|
├── deploy.sh # Apply all manifests to the cluster
|
|
├── .gitignore
|
|
├── helmcharts/ # k3s HelmChart resources (system components)
|
|
│ ├── traefik.yaml # Ingress controller (3 replicas, LoadBalancer)
|
|
│ ├── cert-manager.yaml # TLS certificate manager (Let's Encrypt)
|
|
│ └── longhorn.yaml # Distributed storage (2 replicas, /var/lib/longhorn/)
|
|
├── cluster-config/ # Cluster-wide configuration
|
|
│ ├── namespaces.yaml # Application namespaces (gitea, games, randomly)
|
|
│ ├── clusterissuer.yaml # Let's Encrypt ClusterIssuer
|
|
│ └── image-pull-secret-template.txt # How to create gitea-registry secrets
|
|
└── workloads/ # Application manifests (deploy + svc + ingress + pvc)
|
|
├── gitea/gitea.yaml
|
|
├── wolsey/wolsey.yaml
|
|
├── school-games-prod/school-games-prod.yaml
|
|
├── school-games-dev/school-games-dev.yaml
|
|
├── school-games-default/school-games-default.yaml
|
|
└── randomly/randomly.yaml
|
|
```
|
|
|
|
### How to update a service
|
|
|
|
1. Edit the manifest in `workloads/<service>/<service>.yaml`
|
|
2. Apply: `k3s kubectl apply -f workloads/<service>/<service>.yaml`
|
|
3. Commit and push to keep the repo in sync
|
|
|
|
### How to update a HelmChart (system component)
|
|
|
|
1. Edit the manifest in `helmcharts/<component>.yaml` (e.g. bump `version:` for cert-manager)
|
|
2. Apply: `k3s kubectl apply -f helmcharts/<component>.yaml`
|
|
3. k3s HelmChart controller will automatically reconcile the change
|
|
4. Commit and push
|
|
|
|
### How to deploy from scratch
|
|
|
|
```bash
|
|
# From any k3s node:
|
|
KUBECONFIG=/etc/rancher/k3s/k3s.yaml ./deploy.sh
|
|
```
|