# Agents & Services

Operational reference for all workloads and system services in the imcu.ro cluster.

## System Services

### Netbird VPN

| Property | Value |
|---|---|
| Purpose | WireGuard mesh connecting all k3s nodes |
| Control plane | 167.233.140.254 (netbird.imcu.ro) |
| Interface | wt0 |
| Client version | v0.77.0 |
| Setup key | `E5496A84-6B50-4B13-943A-102B36D5D190` (reusable) |
| Management URL | https://netbird.imcu.ro:443 |
| Local auth | Embedded Dex IdP, password auth |
| TLS | Let's Encrypt |

**Per-node commands:**
```bash
netbird status          # show connection state
netbird up --management-url https://netbird.imcu.ro:443 --hostname <hostname> --disable-client-routes --disable-server-routes --disable-dns
netbird down
```

**VPN IPs:**
- k3s-fi-01: 100.121.136.204
- k3s-de-01: 100.121.191.170
- k3s-de-02: 100.121.49.137

### k3s

| Property | Value |
|---|---|
| Version | v1.36.3+k3s1 |
| Mode | HA (embedded etcd, 3 control-plane nodes) |
| Data dir | /var/lib/rancher/k3s |
| Kubeconfig | /etc/rancher/k3s/k3s.yaml |
| Binaries | k3s, kubectl (symlink), crictl (symlink), ctr (symlink) |
| Service | systemd `k3s.service` |

**Useful commands:**
```bash
k3s kubectl get nodes
k3s kubectl get pods -A
systemctl status k3s
journalctl -u k3s -f
```

### Longhorn

| Property | Value |
|---|---|
| Version | v1.12.1 |
| Namespace | longhorn-system |
| Storage class | longhorn (default) |
| Default replicas | 2 |
| Disk path | /var/lib/longhorn/ |
| UI | ClusterIP 10.43.255.96:80 (SSH tunnel only) |
| Managed via | k3s HelmChart in kube-system |

**HelmChart values:**
```yaml
defaultSettings:
  defaultDataPath: /var/lib/longhorn/
  defaultReplicaCount: 2
persistence:
  defaultClass: true
  defaultClassReplicaCount: 2
```

**Accessing the UI:**
```bash
ssh -L 8080:10.43.255.96:80 root@89.167.46.12
# Open http://localhost:8080
```

**Adding a disk to a node:**
1. Attach new volume in Hetzner Cloud
2. Mount it: `mount /dev/disk/by-id/scsi-<id> /mnt/longhorn-disk2`
3. Add to `/etc/fstab` for persistence
4. In Longhorn UI: Node > Edit > Add Disk (path `/mnt/longhorn-disk2`)

### Traefik

| Property | Value |
|---|---|
| Namespace | kube-system |
| Replicas | 3 (one per node) |
| Service type | LoadBalancer (external IPs: all 3 node public IPs) |
| IngressClass | traefik (default) |
| Managed via | k3s HelmChart in kube-system |

**Ports:**
- 80/TCP (HTTP, redirects to HTTPS)
- 443/TCP (HTTPS)

### cert-manager

| Property | Value |
|---|---|
| Version | v1.18.2 |
| Namespace | cert-manager |
| ClusterIssuer | letsencrypt-prod |
| ACME server | https://acme-v02.api.letsencrypt.org/directory |
| Email | admin@imcu.ro |
| Challenge | HTTP-01 via Traefik ingress |
| Managed via | k3s HelmChart in kube-system |

**All ingresses must have:**
```yaml
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
```

**Check cert status:**
```bash
k3s kubectl get certificate -A
k3s kubectl describe certificate <name> -n <namespace>
```

## Application Workloads

### Gitea

| Property | Value |
|---|---|
| Namespace | gitea |
| Image | gitea/gitea:1.27.2-rootless |
| URL | https://gitea.imcu.ro |
| SSH | gitea.imcu.ro:30022 (NodePort) |
| Database | SQLite (on Longhorn PVC) |
| Storage | 10 Gi Longhorn PVC `gitea-data` |
| Admin user | mcimpoesu (mihai@imcu.ro) |
| Registry | Built-in (enabled) |
| Packages | Enabled |

**Container registry:**
```bash
docker login gitea.imcu.ro
# Username: mcimpoesu
# Password: <Gitea access token>
docker push gitea.imcu.ro/homeschool/<image>:<tag>
```

**Image repositories in Gitea:**
- homeschool/games (tags: 2a4f03df, 7d3a57d)
- homeschool/school (tag: e76d244)
- homeschool/school-games (tag: 20260112T132336)
- homeschool/randomly (tag: latest)

**Create access token:**
```bash
k3s kubectl exec -n gitea deploy/gitea -- gitea admin user generate-access-token \
  --username mcimpoesu --token-name "<name>" --scopes "read:repository,write:package,read:package"
```

**Gitea config location:** `/etc/gitea/app.ini` inside the container (mounted from PVC `config` subPath).

### Wolsey

| Property | Value |
|---|---|
| Namespace | default |
| Image | gitea.imcu.ro/homeschool/school:e76d244 |
| URL | https://wolsey.imcu.ro |
| Storage | 10 Gi Longhorn PVC `wolsey-content` (mounted at /content, readOnly) |
| Content size | 6.5 GB (markdown-based educational content) |
| Port | 3000 |

**Environment:**
- PORT=3000
- HOSTNAME=0.0.0.0
- NODE_ENV=production
- CONTENT_ROOT=/content

### School Games (production)

| Property | Value |
|---|---|
| Namespace | games |
| Image | gitea.imcu.ro/homeschool/school-games:20260112T132336 |
| URL | https://school.imcu.ro |
| Port | 3000 |

### School Games (develop)

| Property | Value |
|---|---|
| Namespace | default |
| Image | gitea.imcu.ro/homeschool/games:7d3a57d |
| URL | https://develop.imcu.ro |
| Port | 3000 |

### School Games (default)

| Property | Value |
|---|---|
| Namespace | default |
| Image | gitea.imcu.ro/homeschool/games:2a4f03df |
| URL | none (internal service only) |
| Port | 3000 |

### Randomly

| Property | Value |
|---|---|
| Namespace | randomly |
| Image | gitea.imcu.ro/homeschool/randomly:latest |
| URL | https://randomly.imcu.ro |
| Port | 3000 |

**Environment:**
- PORT=3000
- HOSTNAME=0.0.0.0
- NODE_ENV=production

## Image Pull Secrets

All namespaces pulling from Gitea registry use secret `gitea-registry`:

```bash
k3s kubectl create secret docker-registry gitea-registry \
  --namespace=<ns> \
  --docker-server=gitea.imcu.ro \
  --docker-username=mcimpoesu \
  --docker-password=<token> \
  --docker-email=mihai@imcu.ro
```

Present in namespaces: `default`, `games`, `randomly`.

## School (decommissioned)

| Property | Value |
|---|---|
| IP | 157.90.228.70 |
| Hostname | school |
| k3s | v1.36.3+k3s1 (still running, system pods only) |
| Status | All workloads migrated, data preserved as backup |

**Backup data on school:**
- `/opt/gitea/` — 255 MB (Gitea data)
- `/opt/registry-data/` — 411 MB (old standalone registry)
- `/mnt/HC_Volume_106055137/school-content/` — 6.5 GB (Wolsey content)

School is reachable via SSH from all k3s nodes (public IP) but is not on the VPN.

## Cloudflare DNS

| Property | Value |
|---|---|
| Zone | imcu.ro |
| Zone ID | 0fca28c6b103c6a730f5c34de3543503 |
| Nameservers | neil.ns.cloudflare.com, may.ns.cloudflare.com |
| API token | Stored in environment as CF_API_TOKEN (Zone:DNS:Edit permission) |

**Manage DNS via API:**
```bash
export CF_API_TOKEN="<token>"
# List records
curl -s -X GET "https://api.cloudflare.com/client/v4/zones/0fca28c6b103c6a730f5c34de3543503/dns_records?per_page=100" \
  -H "Authorization: Bearer $CF_API_TOKEN" | python3 -m json.tool
```

## Common Operations

### Deploy a new service

1. Create namespace if needed: `k3s kubectl create ns <name>`
2. Create image pull secret if pulling from Gitea registry
3. Apply deployment + service manifest
4. Apply ingress with cert-manager annotation:
   ```yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       cert-manager.io/cluster-issuer: letsencrypt-prod
   spec:
     ingressClassName: traefik
     tls:
     - hosts: [<domain>]
       secretName: <domain>-tls
   ```
5. Add DNS A records (3x, one per node IP) via Cloudflare API

### Add a new node to the cluster

1. Provision Hetzner server with Ubuntu
2. Install Netbird: `curl -fsSL https://pkgs.netbird.io/install.sh | sh`
3. Enroll: `netbird login --management-url https://netbird.imcu.ro:443 --setup-key E5496A84-6B50-4B13-943A-102B36D5D190 --hostname <name>`
4. Bring up tunnel: `netbird up --management-url https://netbird.imcu.ro:443 --hostname <name> --disable-client-routes --disable-server-routes --disable-dns`
5. Mount Hetzner volume at `/var/lib/longhorn`, update fstab
6. Install k3s:
   ```bash
   curl -sfL https://get.k3s.io | sh -s - server \
     --server https://100.121.136.204:6443 \
     --token <token-from-/var/lib/rancher/k3s/server/node-token> \
     --node-ip <vpn-ip> \
     --node-external-ip <public-ip> \
     --flannel-iface wt0 \
     --disable traefik
   ```
7. Distribute SSH keys

### Back up Gitea

```bash
# Dump Gitea data
k3s kubectl exec -n gitea deploy/gitea -- gitea dump -c /etc/gitea/app.ini
# Copy dump out
k3s kubectl cp gitea/<pod>:/data/gitea-dump-*.zip ./gitea-backup.zip
```

### Update a deployment image

```bash
# Push new image to Gitea registry
docker push gitea.imcu.ro/homeschool/<image>:<new-tag>

# Update deployment
k3s kubectl set image deploy/<name> <container>=gitea.imcu.ro/homeschool/<image>:<new-tag> -n <namespace>
```