feat(longhorn): expose UI through Envoy Gateway with Keycloak OIDC

[tylerpotts]

Summary

Exposes the Longhorn UI at https://longhorn.<domain> through the existing Envoy Gateway, gated by Keycloak OIDC via an Envoy Gateway SecurityPolicy, and restricted to members of the longhorn-admins Keycloak group. Routes / policies / cert SAN / Keycloak client are only provisioned when the provider has Longhorn enabled (AWS / Hetzner default-on; Azure / GCP / Local off).

• Per-provider gate: new provider.InfraSettings.LonghornEnabled populated from existing per-provider LonghornEnabled() config methods. Threaded through TemplateData to gate manifest rendering.
• Dual-namespace OIDC secret: foundational install writes longhorn-oidc-client-secret into both keycloak (consumed by realm-setup-job to register the OIDC client) and longhorn-system (consumed by the SecurityPolicy). No ReferenceGrant needed.
• New manifests (all conditionally rendered on LonghornEnabled):
  • manifests/networking/routes/longhorn-httproute.yaml — HTTPRoute longhorn in longhorn-system → longhorn-frontend:8080, attaches to nebari-gateway/https
  • manifests/networking/policies/longhorn-securitypolicy.yaml — SecurityPolicy longhorn-oidc enforcing Keycloak OIDC (clientID longhorn, redirect https://longhorn.<domain>/oauth2/callback), plus a JWT provider and an authorization rule that Allows only when the groups claim contains longhorn-admins (everything else 403)
  • New ArgoCD Application securitypolicies so the new policies/ directory is reconciled
• Cert SAN: gateway-certificate.yaml adds longhorn.<domain> to dnsNames when enabled.
• Keycloak realm setup: realm-setup-job.yaml registers the longhorn OIDC client, creates longhorn-admins / longhorn-viewers groups, attaches groups scope, and adds the realm admin to longhorn-admins.

Authorization gates the UI to members of the longhorn-admins Keycloak group. Non-members get HTTP 403 from the gateway. The longhorn-viewers group is created but currently a no-op (Longhorn has no read-only mode).

Test Plan

Unit tests added (all passing under make check + go test -race ./...):

• TestProvider_InfraSettings_LonghornEnabled on AWS and Hetzner (default + explicit-disabled)
• TestCreateLonghornSecrets — dual-namespace Secret creation; no-op when ClientSecret empty
• TestCreateKeycloakAndLonghornSecrets_Together — coexistence with the existing ArgoCD secret
• TestWriteAllToGit_LonghornHTTPRoute — positive + negative render
• TestWriteAllToGit_LonghornSecurityPolicy — positive + negative render; positive asserts forwardAccessToken, jwt provider, authorization Deny-by-default + Allow-longhorn-admins rule
• TestWriteAllToGit_GatewayCertIncludesLonghorn — SAN added/omitted
• TestWriteAllToGit_RealmSetupRegistersLonghornClient — kcadm block + env var renders/omits

Operator-driven acceptance (NOT automated; run after deploy):

• Deploy with domain: <real-domain> + Longhorn enabled (default for AWS): ./nic deploy --config ...
• kubectl -n envoy-gateway-system get certificate nebari-gateway-cert -o jsonpath='{.spec.dnsNames}' includes longhorn.<domain>
• kubectl -n longhorn-system get httproute longhorn shows Accepted=True
• kubectl -n longhorn-system get securitypolicy longhorn-oidc shows Accepted=True
• kubectl -n keycloak logs job/keycloak-realm-setup | grep -i longhorn shows the new client + groups created
• Sign in as the realm admin user (already in longhorn-admins) → land on the Longhorn UI
• Create a fresh Keycloak user, do NOT add to longhorn-admins, sign in → expect HTTP 403 from the gateway
• Add that user to longhorn-admins via the Keycloak UI, sign out and back in → land on the Longhorn UI
• Set aws.longhorn.enabled: false, redeploy → HTTPRoute / SecurityPolicy / Keycloak client / cert SAN all absent

Follow-ups (out of scope)

• Client-secret rotation on redeploy (same pre-existing divergence risk as today's ArgoCD client)

[viniciusdc]

I am looking at this @tylerpotts, thanks

[dcmcand]

Thanks for this - the architecture is sound and the enabled path is solid, well-tested work. Requesting changes on one Blocker around the disabled/teardown path.

Highlights

• Provider consistency is excellent. AWS/Hetzner track Config.LonghornEnabled() (default-on); Azure/GCP/local hardcode false with clear not yet wired comments. The default-then-refine pattern matches the existing StorageClass handling exactly, and LonghornEnabled flows through provider.InfraSettings -> TemplateData with no provider-name switches anywhere. Textbook use of the abstraction boundary.
• Dual-namespace secret handling is clean (createLonghornSecrets): writes the OIDC client secret to both keycloak (for realm-setup-job) and longhorn-system (for the SecurityPolicy), with a comment explaining why, and no-ops on empty.
• generateSecurePassword is reused, not reinvented - crypto rand.Reader, generated only when LonghornEnabled.
• Strong test coverage: both enabled and disabled branches asserted for the HTTPRoute, SecurityPolicy, realm-setup-job, and gateway cert.
• The group-claim path-form comment documents a real cross-repo contract with nebari-operator instead of silently working around it - good defensive documentation.
• No shell-injection risk in realm-setup-job: $LONGHORN_CLIENT_SECRET comes from a secretKeyRef and is quoted.

Blocker

The securitypolicies Application is created unconditionally, but its only manifest is conditionally empty - this diverges from the MetalLB precedent and breaks the Longhorn-disable path.

apps/securitypolicies.yaml has no LonghornEnabled gate and uses allowEmpty: false, while manifests/networking/policies/ contains only longhorn-securitypolicy.yaml, which renders empty when Longhorn is disabled.

The established pattern for "a provider doesn't need this" is MetalLB: writer.go (isMetalLBPath) skips writing both apps/metallb*.yaml and manifests/metallb*, so no Application is ever created. This PR instead leaves the securitypolicies Application pointing at a directory that yields zero resources whenever Longhorn is off (Azure, GCP, local - the primary dev/test path - and AWS/Hetzner with longhorn_enabled: false).

Concrete failure mode: with allowEmpty: false, when a cluster goes from Longhorn-enabled to disabled (config toggle + redeploy), ArgoCD refuses to prune the SecurityPolicy to zero, leaving an orphaned policy and an OutOfSync app that selfHeal cannot reconcile.

Suggested fix: follow the MetalLB precedent - generalize the skip mechanism (e.g. an isLonghornPath check or a small skip-list) so that when !settings.LonghornEnabled the writer skips both apps/securitypolicies.yaml and manifests/networking/policies/longhorn-securitypolicy.yaml. That removes the empty committed file and the orphan-on-disable problem in one move.

One thing I could not verify without a live ArgoCD: the exact sync status of a fresh non-Longhorn deploy (target=0, live=0) - it may show Synced/Healthy harmlessly rather than erroring. Can you confirm what you observed on a local/Azure deploy? Even if the fresh case is benign, the toggle-off orphan case and the divergence from the deliberate MetalLB skip pattern still stand.

Question

deploy.go gating vs Keycloak: longhornClientSecret is generated on infraSettings.LonghornEnabled alone, but createLonghornSecrets only runs inside if foundationalCfg.Keycloak.Enabled. The writer comment says Keycloak is mandatory infrastructure, so LonghornEnabled && !Keycloak.Enabled shouldn't occur - is that invariant guaranteed, or relying on deploy.go always setting Keycloak.Enabled: true? If the latter, a one-line comment at the generation site would prevent a future dangling secretKeyRef.