cluster/k3s-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: imcu.ro HA k3s cluster manifests

Browse Source 
- 3-node HA cluster (k3s v1.36.3+k3s1, embedded etcd) on Hetzner
- Netbird VPN mesh (wt0) for all cluster traffic
- Longhorn distributed storage (2 replicas, /var/lib/longhorn/)
- Traefik ingress (3 replicas, LoadBalancer on all node IPs)
- cert-manager with Let's Encrypt (auto-renewing TLS)
- Gitea (git + container registry) with SQLite on Longhorn
- 4 migrated workloads: wolsey, school-games (3 variants), randomly
- All HelmCharts and workload manifests as YAML
- deploy.sh for one-shot cluster setup
This commit is contained in:
mcimpoesu
2026-08-16 21:35:21 +03:00
commit fd7c529255
16 changed files with 1439 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+326
View File
@@ -0,0 +1,326 @@
# Agents & Services
Operational reference for all workloads and system services in the imcu.ro cluster.
## System Services
### Netbird VPN
| Property | Value |
|---|---|
| Purpose | WireGuard mesh connecting all k3s nodes |
| Control plane | 167.233.140.254 (netbird.imcu.ro) |
| Interface | wt0 |
| Client version | v0.77.0 |
| Setup key | `E5496A84-6B50-4B13-943A-102B36D5D190` (reusable) |
| Management URL | https://netbird.imcu.ro:443 |
| Local auth | Embedded Dex IdP, password auth |
| TLS | Let's Encrypt |
**Per-node commands:**
```bash
netbird status # show connection state
netbird up --management-url https://netbird.imcu.ro:443 --hostname <hostname> --disable-client-routes --disable-server-routes --disable-dns
netbird down
```
**VPN IPs:**
- k3s-fi-01: 100.121.136.204
- k3s-de-01: 100.121.191.170
- k3s-de-02: 100.121.49.137
### k3s
| Property | Value |
|---|---|
| Version | v1.36.3+k3s1 |
| Mode | HA (embedded etcd, 3 control-plane nodes) |
| Data dir | /var/lib/rancher/k3s |
| Kubeconfig | /etc/rancher/k3s/k3s.yaml |
| Binaries | k3s, kubectl (symlink), crictl (symlink), ctr (symlink) |
| Service | systemd `k3s.service` |
**Useful commands:**
```bash
k3s kubectl get nodes
k3s kubectl get pods -A
systemctl status k3s
journalctl -u k3s -f
```
### Longhorn
| Property | Value |
|---|---|
| Version | v1.12.1 |
| Namespace | longhorn-system |
| Storage class | longhorn (default) |
| Default replicas | 2 |
| Disk path | /var/lib/longhorn/ |
| UI | ClusterIP 10.43.255.96:80 (SSH tunnel only) |
| Managed via | k3s HelmChart in kube-system |
**HelmChart values:**
```yaml
defaultSettings:
defaultDataPath: /var/lib/longhorn/
defaultReplicaCount: 2
persistence:
defaultClass: true
defaultClassReplicaCount: 2
```
**Accessing the UI:**
```bash
ssh -L 8080:10.43.255.96:80 root@89.167.46.12
# Open http://localhost:8080
```
**Adding a disk to a node:**
1. Attach new volume in Hetzner Cloud
2. Mount it: `mount /dev/disk/by-id/scsi-<id> /mnt/longhorn-disk2`
3. Add to `/etc/fstab` for persistence
4. In Longhorn UI: Node > Edit > Add Disk (path `/mnt/longhorn-disk2`)
### Traefik
| Property | Value |
|---|---|
| Namespace | kube-system |
| Replicas | 3 (one per node) |
| Service type | LoadBalancer (external IPs: all 3 node public IPs) |
| IngressClass | traefik (default) |
| Managed via | k3s HelmChart in kube-system |
**Ports:**
- 80/TCP (HTTP, redirects to HTTPS)
- 443/TCP (HTTPS)
### cert-manager
| Property | Value |
|---|---|
| Version | v1.18.2 |
| Namespace | cert-manager |
| ClusterIssuer | letsencrypt-prod |
| ACME server | https://acme-v02.api.letsencrypt.org/directory |
| Email | admin@imcu.ro |
| Challenge | HTTP-01 via Traefik ingress |
| Managed via | k3s HelmChart in kube-system |
**All ingresses must have:**
```yaml
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
```
**Check cert status:**
```bash
k3s kubectl get certificate -A
k3s kubectl describe certificate <name> -n <namespace>
```
## Application Workloads
### Gitea
| Property | Value |
|---|---|
| Namespace | gitea |
| Image | gitea/gitea:1.27.2-rootless |
| URL | https://gitea.imcu.ro |
| SSH | gitea.imcu.ro:30022 (NodePort) |
| Database | SQLite (on Longhorn PVC) |
| Storage | 10 Gi Longhorn PVC `gitea-data` |
| Admin user | mcimpoesu (mihai@imcu.ro) |
| Registry | Built-in (enabled) |
| Packages | Enabled |
**Container registry:**
```bash
docker login gitea.imcu.ro
# Username: mcimpoesu
# Password: <Gitea access token>
docker push gitea.imcu.ro/homeschool/<image>:<tag>
```
**Image repositories in Gitea:**
- homeschool/games (tags: 2a4f03df, 7d3a57d)
- homeschool/school (tag: e76d244)
- homeschool/school-games (tag: 20260112T132336)
- homeschool/randomly (tag: latest)
**Create access token:**
```bash
k3s kubectl exec -n gitea deploy/gitea -- gitea admin user generate-access-token \
--username mcimpoesu --token-name "<name>" --scopes "read:repository,write:package,read:package"
```
**Gitea config location:** `/etc/gitea/app.ini` inside the container (mounted from PVC `config` subPath).
### Wolsey
| Property | Value |
|---|---|
| Namespace | default |
| Image | gitea.imcu.ro/homeschool/school:e76d244 |
| URL | https://wolsey.imcu.ro |
| Storage | 10 Gi Longhorn PVC `wolsey-content` (mounted at /content, readOnly) |
| Content size | 6.5 GB (markdown-based educational content) |
| Port | 3000 |
**Environment:**
- PORT=3000
- HOSTNAME=0.0.0.0
- NODE_ENV=production
- CONTENT_ROOT=/content
### School Games (production)
| Property | Value |
|---|---|
| Namespace | games |
| Image | gitea.imcu.ro/homeschool/school-games:20260112T132336 |
| URL | https://school.imcu.ro |
| Port | 3000 |
### School Games (develop)
| Property | Value |
|---|---|
| Namespace | default |
| Image | gitea.imcu.ro/homeschool/games:7d3a57d |
| URL | https://develop.imcu.ro |
| Port | 3000 |
### School Games (default)
| Property | Value |
|---|---|
| Namespace | default |
| Image | gitea.imcu.ro/homeschool/games:2a4f03df |
| URL | none (internal service only) |
| Port | 3000 |
### Randomly
| Property | Value |
|---|---|
| Namespace | randomly |
| Image | gitea.imcu.ro/homeschool/randomly:latest |
| URL | https://randomly.imcu.ro |
| Port | 3000 |
**Environment:**
- PORT=3000
- HOSTNAME=0.0.0.0
- NODE_ENV=production
## Image Pull Secrets
All namespaces pulling from Gitea registry use secret `gitea-registry`:
```bash
k3s kubectl create secret docker-registry gitea-registry \
--namespace=<ns> \
--docker-server=gitea.imcu.ro \
--docker-username=mcimpoesu \
--docker-password=<token> \
--docker-email=mihai@imcu.ro
```
Present in namespaces: `default`, `games`, `randomly`.
## School (decommissioned)
| Property | Value |
|---|---|
| IP | 157.90.228.70 |
| Hostname | school |
| k3s | v1.36.3+k3s1 (still running, system pods only) |
| Status | All workloads migrated, data preserved as backup |
**Backup data on school:**
- `/opt/gitea/` — 255 MB (Gitea data)
- `/opt/registry-data/` — 411 MB (old standalone registry)
- `/mnt/HC_Volume_106055137/school-content/` — 6.5 GB (Wolsey content)
School is reachable via SSH from all k3s nodes (public IP) but is not on the VPN.
## Cloudflare DNS
| Property | Value |
|---|---|
| Zone | imcu.ro |
| Zone ID | 0fca28c6b103c6a730f5c34de3543503 |
| Nameservers | neil.ns.cloudflare.com, may.ns.cloudflare.com |
| API token | Stored in environment as CF_API_TOKEN (Zone:DNS:Edit permission) |
**Manage DNS via API:**
```bash
export CF_API_TOKEN="<token>"
# List records
curl -s -X GET "https://api.cloudflare.com/client/v4/zones/0fca28c6b103c6a730f5c34de3543503/dns_records?per_page=100" \
-H "Authorization: Bearer $CF_API_TOKEN" | python3 -m json.tool
```
## Common Operations
### Deploy a new service
1. Create namespace if needed: `k3s kubectl create ns <name>`
2. Create image pull secret if pulling from Gitea registry
3. Apply deployment + service manifest
4. Apply ingress with cert-manager annotation:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts: [<domain>]
secretName: <domain>-tls
```
5. Add DNS A records (3x, one per node IP) via Cloudflare API
### Add a new node to the cluster
1. Provision Hetzner server with Ubuntu
2. Install Netbird: `curl -fsSL https://pkgs.netbird.io/install.sh | sh`
3. Enroll: `netbird login --management-url https://netbird.imcu.ro:443 --setup-key E5496A84-6B50-4B13-943A-102B36D5D190 --hostname <name>`
4. Bring up tunnel: `netbird up --management-url https://netbird.imcu.ro:443 --hostname <name> --disable-client-routes --disable-server-routes --disable-dns`
5. Mount Hetzner volume at `/var/lib/longhorn`, update fstab
6. Install k3s:
```bash
curl -sfL https://get.k3s.io | sh -s - server \
--server https://100.121.136.204:6443 \
--token <token-from-/var/lib/rancher/k3s/server/node-token> \
--node-ip <vpn-ip> \
--node-external-ip <public-ip> \
--flannel-iface wt0 \
--disable traefik
```
7. Distribute SSH keys
### Back up Gitea
```bash
# Dump Gitea data
k3s kubectl exec -n gitea deploy/gitea -- gitea dump -c /etc/gitea/app.ini
# Copy dump out
k3s kubectl cp gitea/<pod>:/data/gitea-dump-*.zip ./gitea-backup.zip
```
### Update a deployment image
```bash
# Push new image to Gitea registry
docker push gitea.imcu.ro/homeschool/<image>:<new-tag>
# Update deployment
k3s kubectl set image deploy/<name> <container>=gitea.imcu.ro/homeschool/<image>:<new-tag> -n <namespace>
```