Wardyn k8s setup — cluster prereqs, values, Entra RBAC, install, verify

Goal: get wardynd running on Kubernetes with the k8s runner substrate
actually able to create confined sandboxes, and admin/member RBAC wired
through a real IdP (Entra ID is the worked example — the trap it has that a
generic OIDC provider doesn't is exactly why it gets its own section below).
Reuse the shipped chart and substrate; never hand-roll a manifest.

Ground truth (read these, don't restate from memory)

• Chart defaults + every value's own doc comment: deploy/helm/wardyn/values.yaml.
• Worked install commands, the k8s.enabled walkthrough, and the honest
  gap list (what the k8s substrate does not do yet — BYOI, local_dir
  mounts, per-pod PIDs/disk limits, ground-truth correlator): deploy/helm/wardyn/README.md.
• Every WARDYN_* var this touches, with defaults and gotchas spelled out
  per row (WARDYN_K8S_*, WARDYN_OIDC_*, WARDYN_SSH_*, WARDYN_TLS_TERMINATED): docs/ENV.md.
• Multi-user semantics (admin vs member, ownership scoping, the shared
  admin-token ceiling) and the k8s known-gaps detail behind the chart
  README's summary: docs/OPERATIONS.md.
• SSH gateway setup/use once the cluster is up (the run's owner, or an
  admin on a run with no personal owner — the admin override re-checks the key's
  role every WARDYN_SSH_ROLE_TTL;
  see this skill's troubleshooting table): docs/SSH.md.

Recipe

1. Cluster prereqs — prove the CNI enforces NetworkPolicy before anything
   else. The k8s substrate's boot-time egress canary
   (internal/runner/k8s/canary.go) runs a two-phase check against the
   cluster itself the moment wardynd starts with k8s.enabled — a
   throwaway pod first confirms it CAN reach the apiserver with no policy in
   place, then a deny-all NetworkPolicy is applied and the same pod is
   proven UNREACHABLE. If phase two still gets through, the CNI does not
   enforce NetworkPolicy and wardynd refuses to boot the substrate —
   this is fail-closed by design, not a bug to work around. Prove it on a
   throwaway cluster before touching a real one, using the exact recipe CI
   uses (.github/workflows/ci.yml's conformance-k8s job,
   deploy/kind/conformance-kind-config.yaml): kind's bundled kindnet CNI
   does not enforce NetworkPolicy, so start kind with
   networking.disableDefaultCNI: true and install a real CNI yourself —
   Calico is what CI pins:

   kind create cluster --config deploy/kind/conformance-kind-config.yaml
   kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
   kubectl -n kube-system rollout status daemonset/calico-node --timeout=180s

   Any NetworkPolicy-enforcing CNI works the same way on a real cluster
   (Calico, Cilium, etc.) — this is just the reproducible local proof.
   The opt-out and its cost: WARDYN_K8S_ALLOW_UNENFORCED_NETPOL=1
   (helm: env.WARDYN_K8S_ALLOW_UNENFORCED_NETPOL) lets wardynd boot anyway
   on a CNI that doesn't enforce policy — logged loudly at construction, and
   every sandbox this substrate creates then has UNCONFINED egress. Only
   ever accept that on a cluster you have some other confinement story for;
   it is never an invisible downgrade (ClassSupport.NetworkPolicy/
   StructuralEgress both keep reporting false, so /healthz never reads
   like a genuinely confined install).

2. Author values.yaml.

   • Runner: k8s.enabled=true and serviceAccount.automount=true — the
     chart REFUSES to render k8s.enabled without the automount flip (the
     substrate drives the apiserver directly via client-go, which needs the
     pod's own projected token).
   • Images: k8s.proxyImage (required — the chart refuses to render
     with k8s.enabled and no k8s.proxyImage, templates/deployment.yaml; also
     what the egress canary itself launches) and image.repository/image.tag
     for wardynd.
   • runtimeClasses: k8s.runtimeClasses.CC2/.CC3 pin a Confinement
     Class to a RuntimeClass NAME already registered in the cluster (e.g.
     CC2: gvisor) — unlike Docker's well-known runtime family names, a
     RuntimeClass object name is operator-chosen, so CC2/CC3 stay
     unadvertised (CC1-only) until pinned here.
   • SSO block: env.WARDYN_OIDC_ISSUER, env.WARDYN_OIDC_INTERNAL_ISSUER
     (only if the browser and wardynd reach the IdP at different hostnames),
     and extraEnv for WARDYN_OIDC_CLIENT_SECRET (🔒 in docs/ENV.md — never
     a literal env value). RBAC itself is env.WARDYN_OIDC_ROLE_MAP — see
     step 3.
   • SSH block: ssh.enabled, ssh.port (default 2222), and
     ssh.advertiseHost (required when enabled — the chart never guesses a
     public address). See Split SSH exposure
     if SSH needs different exposure than HTTP (its own LoadBalancer while
     HTTP stays ClusterIP, for example).

3. Entra ID app registration — admin/member RBAC. Wardyn has real
   admin/user roles (internal/auth/oidc's deriveRole), and Entra
   App Roles are the recommended way to feed them — do this as its own step,
   not folded into general SSO setup, because Entra has a trap a generic OIDC
   provider doesn't (below).

   • Register the App Roles on the app registration (Azure Portal →
     App registrations → your app → App roles → Create app role, or via
     the manifest editor):

     {
       "appRoles": [
         {
           "id": "3b1f6e2a-...-generate-a-fresh-guid",
           "allowedMemberTypes": ["User"],
           "displayName": "Wardyn Admin",
           "value": "Wardyn.Admin",
           "description": "Full Wardyn control-plane access",
           "isEnabled": true
         },
         {
           "id": "9c2d7f4b-...-generate-a-fresh-guid",
           "allowedMemberTypes": ["User"],
           "displayName": "Wardyn Member",
           "value": "Wardyn.Member",
           "description": "Launch and use Wardyn runs",
           "isEnabled": true
         }
       ]
     }

     (the appRoles key merges into the app's existing manifest — the
     snippet above is deliberately just that one key, not a full manifest
     replacement)
     value is what lands in the ID token's roles claim — that string is
     what WARDYN_OIDC_ROLE_MAP matches against, not displayName.
   • Set "Assignment required?" to Yes on the Enterprise Application
     (Azure Portal → Enterprise applications → your app → Properties). Without
     this, Entra hands out the app to any tenant user with NO role assigned —
     the token carries an empty roles claim, and every such login falls
     through to WARDYN_OIDC_DEFAULT_ROLE (or is denied if that's unset). Turn
     it on, then assign each role to the right users/groups under Users and
     groups.
   • The role-map values — env.WARDYN_OIDC_ROLE_MAP maps the App Role's
     value (or a groups entry, or an email) to a Wardyn role:

     env:
       WARDYN_OIDC_ROLE_MAP: "Wardyn.Admin=admin,Wardyn.Security=security_admin,Wardyn.Member=user"
       # Optional: no match at all -> this role instead of denying the login.
       WARDYN_OIDC_DEFAULT_ROLE: "user"

     security_admin is reachable only through this map — there is no
     allowlist twin of WARDYN_OIDC_OPERATOR_EMAILS for it, and
     WARDYN_OIDC_DEFAULT_ROLE refuses security_admin at boot (it accepts
     only admin/user).
     Matching is case-insensitive and ASCII-only; any admin match wins over
     a user match regardless of which claim produced it; an email on
     WARDYN_OIDC_OPERATOR_EMAILS is an ADDITIONAL admin match (see step 3's
     legacy note below). Leaving WARDYN_OIDC_ROLE_MAP unset disables role
     derivation entirely — every signed-in human is admin, upgrade-safe but
     not RBAC.
   • The email_verified trap: Entra ID tokens typically omit
     email_verified entirely. WARDYN_OIDC_EMAIL_DOMAINS (domain-restricted
     login) fails closed on that — it denies every login against an Entra
     tenant, since it requires a verified email that never arrives. Do not
     use WARDYN_OIDC_EMAIL_DOMAINS with Entra; App Roles plus "assignment
     required" is the tenant-scoping mechanism instead — only assigned users
     ever reach a role, and everyone else denies via WARDYN_OIDC_DEFAULT_ROLE
     (or the same fall-through denial with it unset).
   • Legacy allowlist still works: WARDYN_OIDC_OPERATOR_EMAILS (the
     pre-0.5 operator list) is not replaced by WARDYN_OIDC_ROLE_MAP — an
     email on it is still an additional admin match, so an existing
     deployment adopting App Roles keeps its current operators as admins with
     zero re-configuration.
   • env.WARDYN_OIDC_ROLE_MAP is the chart's BOOTSTRAP layer, not the only
     place mappings get edited. Once the install is up, an admin adds,
     edits and removes further mappings live from the console's Getting
     Started → People step (GET /access, POST /access/mappings, DELETE /access/mappings/{id}) — no chart change or redeploy needed. The chart
     map always wins a duplicate key against a console row; a console row
     takes effect at that person's next sign-in, never retroactively. An
     email-shaped console mapping is refused by default (steer to an App
     Role or groups value instead, per the trap above) — opt in with
     env.WARDYN_OIDC_ALLOW_EMAIL_MAPPINGS=true only when this deployment
     has no usable App Role/groups claim at all. Full semantics (the merged
     table, the posture-flip and lockout guards, the preview panel):
     docs/OPERATIONS.md's "Who decides who gets in".
     A scripted, worked validation of this whole path against a real
     (free-tier, throwaway) Entra tenant lives in deploy/azure-entra-sso/ —
     run it before trusting any of this on a tenant that matters.

4. Install / upgrade.

   kubectl create namespace wardyn
   kubectl create namespace wardyn-runs   # k8s.runsNamespace below — must pre-exist
   
   kubectl create secret generic wardyn-auth -n wardyn \
     --from-literal=admin-token="$(openssl rand -hex 32)"   # fallback if OIDC misconfigures
   kubectl create secret generic wardyn-pg -n wardyn \
     --from-literal=dsn="$PG_DSN" \
     --from-literal=age-key="$(docker run --rm "$REGISTRY/wardynd:$TAG" -gen-age-key)"
   
   helm upgrade --install wardyn ./deploy/helm/wardyn \
     --namespace wardyn \
     --set postgres.dsn.secretRef.name=wardyn-pg \
     --set secrets.ageKeyFromSecret=true \
     --set auth.adminToken.secretRef.name=wardyn-auth \
     --set serviceAccount.automount=true \
     --set k8s.enabled=true \
     --set k8s.runsNamespace=wardyn-runs \
     --set k8s.proxyImage="$REGISTRY/wardyn-proxy:$TAG" \
     --set k8s.runtimeClasses.CC2="<your gVisor RuntimeClass>" \
     -f my-k8s-values.yaml   # the SSO/SSH/runtimeClasses block from step 2-3

   No CC2/CC3 pin and no defaultPolicy override is needed to render this: the
   shipped policy floor is CC1, so a stock install clears it with the bare
   [CC1] the substrate advertises. The --set k8s.runtimeClasses.CC2=… above
   is optional hardening — pin the gVisor (or Kata) RuntimeClass name from step 2
   to advertise a stronger class (a duplicate of the pin in my-k8s-values.yaml
   is harmless). A stronger floor with no matching RuntimeClass fails closed at
   dispatch. postgres.dsn.secretRef is a PERSISTENT DSN, so the age identity riding in
   the SAME Secret (age-key, via -gen-age-key) is required, not optional —
   without it wardynd generates a fresh identity every boot and cannot decrypt
   what the previous boot encrypted (crash-loops on the SECOND restart). Keep
   an admin token secret configured even with OIDC set up — it is the
   recovery path if the Entra role map ever locks everyone out.

5. Verify.

   • kubectl -n wardyn rollout status deploy/wardyn --timeout=120s — not
     ImagePullBackOff (see the troubleshooting table).
   • Console → setup checks: k8s_egress_containment reads Enforcing ·
     NetworkPolicy, not Indeterminate or Not enforcing (see step 1).
   • Launch a run through the console or wardyn run, confirm it reaches
     RUNNING, and attach (web terminal, or SSH if ssh.enabled — ssh <run-id>@<advertiseHost> -p <port>, owner or admin, docs/SSH.md).
   • Sign in via SSO with a member-mapped account and confirm the console
     shows member-scoped nav (no policy/workspace/secret CRUD, no BYOI); sign
     in with an admin-mapped account and confirm the full console.

Troubleshooting