k3s-cluster/AGENTS.md

Agents & Services

Operational reference for all workloads and system services in the imcu.ro cluster.

System Services

Netbird VPN

Property	Value
Purpose	WireGuard mesh connecting all k3s nodes
Control plane	167.233.140.254 (netbird.imcu.ro)
Interface	wt0
Client version	v0.77.0
Setup key	E5496A84-6B50-4B13-943A-102B36D5D190 (reusable)
Management URL	https://netbird.imcu.ro:443
Local auth	Embedded Dex IdP, password auth
TLS	Let's Encrypt

Per-node commands:

netbird status          # show connection state
netbird up --management-url https://netbird.imcu.ro:443 --hostname <hostname> --disable-client-routes --disable-server-routes --disable-dns
netbird down

VPN IPs:

• k3s-fi-01: 100.121.136.204
• k3s-de-01: 100.121.191.170
• k3s-de-02: 100.121.49.137

k3s

Property	Value
Version	v1.36.3+k3s1
Mode	HA (embedded etcd, 3 control-plane nodes)
Data dir	/var/lib/rancher/k3s
Kubeconfig	/etc/rancher/k3s/k3s.yaml
Binaries	k3s, kubectl (symlink), crictl (symlink), ctr (symlink)
Service	systemd k3s.service

Useful commands:

k3s kubectl get nodes
k3s kubectl get pods -A
systemctl status k3s
journalctl -u k3s -f

Longhorn

Property	Value
Version	v1.12.1
Namespace	longhorn-system
Storage class	longhorn (default)
Default replicas	2
Disk path	/var/lib/longhorn/
UI	ClusterIP 10.43.255.96:80 (SSH tunnel only)
Managed via	k3s HelmChart in kube-system

HelmChart values:

defaultSettings:
  defaultDataPath: /var/lib/longhorn/
  defaultReplicaCount: 2
persistence:
  defaultClass: true
  defaultClassReplicaCount: 2

Accessing the UI:

ssh -L 8080:10.43.255.96:80 root@89.167.46.12
# Open http://localhost:8080

Adding a disk to a node:

1. Attach new volume in Hetzner Cloud
2. Mount it: mount /dev/disk/by-id/scsi-<id> /mnt/longhorn-disk2
3. Add to /etc/fstab for persistence
4. In Longhorn UI: Node > Edit > Add Disk (path /mnt/longhorn-disk2)

Traefik

Property	Value
Namespace	kube-system
Replicas	3 (one per node)
Service type	LoadBalancer (external IPs: all 3 node public IPs)
IngressClass	traefik (default)
Managed via	k3s HelmChart in kube-system

Ports:

• 80/TCP (HTTP, redirects to HTTPS)
• 443/TCP (HTTPS)

cert-manager

Property	Value
Version	v1.18.2
Namespace	cert-manager
ClusterIssuer	letsencrypt-prod
ACME server	https://acme-v02.api.letsencrypt.org/directory
Email	admin@imcu.ro
Challenge	HTTP-01 via Traefik ingress
Managed via	k3s HelmChart in kube-system

All ingresses must have:

metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

Check cert status:

k3s kubectl get certificate -A
k3s kubectl describe certificate <name> -n <namespace>

Application Workloads

Gitea

Property	Value
Namespace	gitea
Image	gitea/gitea:1.27.2-rootless
URL	https://gitea.imcu.ro
SSH	gitea.imcu.ro:30022 (NodePort)
Database	SQLite (on Longhorn PVC)
Storage	10 Gi Longhorn PVC gitea-data
Admin user	mcimpoesu (mihai@imcu.ro)
Registry	Built-in (enabled)
Packages	Enabled

Container registry:

docker login gitea.imcu.ro
# Username: mcimpoesu
# Password: <Gitea access token>
docker push gitea.imcu.ro/homeschool/<image>:<tag>

Image repositories in Gitea:

• homeschool/games (tags: 2a4f03df, 7d3a57d)
• homeschool/school (tag: e76d244)
• homeschool/school-games (tag: 20260112T132336)
• homeschool/randomly (tag: latest)

Create access token:

k3s kubectl exec -n gitea deploy/gitea -- gitea admin user generate-access-token \
  --username mcimpoesu --token-name "<name>" --scopes "read:repository,write:package,read:package"

Gitea config location: /etc/gitea/app.ini inside the container (mounted from PVC config subPath).

Wolsey

Property	Value
Namespace	default
Image	gitea.imcu.ro/homeschool/school:e76d244
URL	https://wolsey.imcu.ro
Storage	10 Gi Longhorn PVC wolsey-content (mounted at /content, readOnly)
Content size	6.5 GB (markdown-based educational content)
Port	3000

Environment:

• PORT=3000
• HOSTNAME=0.0.0.0
• NODE_ENV=production
• CONTENT_ROOT=/content

School Games (production)

Property	Value
Namespace	games
Image	gitea.imcu.ro/homeschool/school-games:20260112T132336
URL	https://school.imcu.ro
Port	3000

School Games (develop)

Property	Value
Namespace	default
Image	gitea.imcu.ro/homeschool/games:7d3a57d
URL	https://develop.imcu.ro
Port	3000

School Games (default)

Property	Value
Namespace	default
Image	gitea.imcu.ro/homeschool/games:2a4f03df
URL	none (internal service only)
Port	3000

Randomly

Property	Value
Namespace	randomly
Image	gitea.imcu.ro/homeschool/randomly:latest
URL	https://randomly.imcu.ro
Port	3000

Environment:

• PORT=3000
• HOSTNAME=0.0.0.0
• NODE_ENV=production

Image Pull Secrets

All namespaces pulling from Gitea registry use secret gitea-registry:

k3s kubectl create secret docker-registry gitea-registry \
  --namespace=<ns> \
  --docker-server=gitea.imcu.ro \
  --docker-username=mcimpoesu \
  --docker-password=<token> \
  --docker-email=mihai@imcu.ro

Present in namespaces: default, games, randomly.

School (decommissioned)

Property	Value
IP	157.90.228.70
Hostname	school
k3s	v1.36.3+k3s1 (still running, system pods only)
Status	All workloads migrated, data preserved as backup

Backup data on school:

• /opt/gitea/ — 255 MB (Gitea data)
• /opt/registry-data/ — 411 MB (old standalone registry)
• /mnt/HC_Volume_106055137/school-content/ — 6.5 GB (Wolsey content)

School is reachable via SSH from all k3s nodes (public IP) but is not on the VPN.

Cloudflare DNS

Property	Value
Zone	imcu.ro
Zone ID	0fca28c6b103c6a730f5c34de3543503
Nameservers	neil.ns.cloudflare.com, may.ns.cloudflare.com
API token	Stored in environment as CF_API_TOKEN (Zone:DNS:Edit permission)

Manage DNS via API:

export CF_API_TOKEN="<token>"
# List records
curl -s -X GET "https://api.cloudflare.com/client/v4/zones/0fca28c6b103c6a730f5c34de3543503/dns_records?per_page=100" \
  -H "Authorization: Bearer $CF_API_TOKEN" | python3 -m json.tool

Common Operations

Deploy a new service

1. Create namespace if needed: k3s kubectl create ns <name>
2. Create image pull secret if pulling from Gitea registry
3. Apply deployment + service manifest
4. Apply ingress with cert-manager annotation:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       cert-manager.io/cluster-issuer: letsencrypt-prod
   spec:
     ingressClassName: traefik
     tls:
     - hosts: [<domain>]
       secretName: <domain>-tls
   

5. Add DNS A records (3x, one per node IP) via Cloudflare API

Add a new node to the cluster

1. Provision Hetzner server with Ubuntu
2. Install Netbird: curl -fsSL https://pkgs.netbird.io/install.sh | sh
3. Enroll: netbird login --management-url https://netbird.imcu.ro:443 --setup-key E5496A84-6B50-4B13-943A-102B36D5D190 --hostname <name>
4. Bring up tunnel: netbird up --management-url https://netbird.imcu.ro:443 --hostname <name> --disable-client-routes --disable-server-routes --disable-dns
5. Mount Hetzner volume at /var/lib/longhorn, update fstab
6. Install k3s:

   curl -sfL https://get.k3s.io | sh -s - server \
     --server https://100.121.136.204:6443 \
     --token <token-from-/var/lib/rancher/k3s/server/node-token> \
     --node-ip <vpn-ip> \
     --node-external-ip <public-ip> \
     --flannel-iface wt0 \
     --disable traefik
   

7. Distribute SSH keys

Back up Gitea

# Dump Gitea data
k3s kubectl exec -n gitea deploy/gitea -- gitea dump -c /etc/gitea/app.ini
# Copy dump out
k3s kubectl cp gitea/<pod>:/data/gitea-dump-*.zip ./gitea-backup.zip

Update a deployment image

# Push new image to Gitea registry
docker push gitea.imcu.ro/homeschool/<image>:<new-tag>

# Update deployment
k3s kubectl set image deploy/<name> <container>=gitea.imcu.ro/homeschool/<image>:<new-tag> -n <namespace>